Using resource-specific tables instead of AzureDiagnostics

A synopsis of this post:

1. Azure resource logs explained
2. Explaining the Azure diagnostics mode (current / deprecating mode)
3. Explaining the modernized Resource-specific mode
4. The list of current Azure resources that support resource-specific mode
5. How do we configure and test the new modernized resource-specific mode?

1. Azure resource logs explained

Azure resource logs are platform logs that provide insight into operations that were performed within an Azure resource. The content of resource logs varies according to the Azure service and resource type. Resource logging is not enabled and collected by default. Diagnostic logging needs to be manually enabled. The diagnostic settings need to be enabled on the specific Azure resources to enable each Azure resource to send its resource logs to respective destinations.

The current default destination for sending Azure resource logs is to the Log Analytics workspace AzureDiagnostics table. A new method will be to send Azure resource logs to multiple resource-specific tables for each category of the resource.

2. Explaining the Azure diagnostics mode (current / deprecating mode)

In this current mode, all data from any diagnostic settings is collected in the central AzureDiagnostics table. This legacy method is used today by most Azure services. Because multiple resource types send data to the same table, its schema is the superset of the schemas for all of the different data types being collected.

The AzureDiagnostics Table

The AzureDiagnostics table is a central storage table in the Log Analytics Workspace which stores resource logs for multiple Azure services that have the Azure Diagnostics enabled. Resource logs describe the internal operations of Azure resources.

Sending data to the legacy Log Analytics Workspace AzureDiagnostics table

Sending resource logs to a Log Analytics workspace is the current method of enabling the features of Azure Monitor Logs on a resource, where you can:

• Correlate resource log data with other monitoring data collected by Azure Monitor.
• Consolidate log entries from multiple Azure resources, subscriptions, and tenants into one location for analysis together.
• Use log queries to perform complex analysis and gain deep insights on log data.
• Use log alerts with complex alerting logic.

Enable diagnostic settings to send resource logs to a Log Analytics workspace. This data is stored in tables as described in Structure of Azure Monitor Logs. The tables used by resource logs depend on what type of collection the resource is using:

• Azure diagnostics: All data is written to the single AzureDiagnostics table (current / legacy methodology)
• Resource-specific: Data is written to multiple individual tables for each category of the resource (new methodology)

3. Explaining the modernized Resource-specific mode

In this new modernized mode, multiple individual tables are provisioned in the selected workspace for each category selected in the diagnostic setting. Microsoft recommends this method going forward because:

• Simplified means to work with the data in log queries.
• Provides improved discoverability of schemas and their structure.
• Improves performance across ingestion latency and query times.
• Provides the ability to grant Azure role-based access control rights on a specific table.
• All Azure services will eventually migrate to the resource-specific mode!

Certain Azure services may be configured to use either the resource-specific mode to store data in multiple tables specific to that service, or the legacy Azure diagnostics mode. When configured in resource-specific mode, the resources cannot send logs to the AzureDiagnostics table.

4. The list of current Azure resources that support resource-specific mode

The Resource Types list below shows the current Azure services that are able to send their logs to resource-specific tables. This list will be incrementally updated in the future as more Azure services / resources are modernized / enabled with the resource-specific logging mode, so keep monitoring this list.

The current list of resource-specific mode enabled Azure resources at the time of writing this blog are:

1. API Management Services
2. Azure Cosmos DB
3. Data factories (V2)
4. IoT Hub
5. Recovery Services vaults(Backup)

Keep monitoring the Microsoft link below for updates to the Azure services list:

https://learn.microsoft.com/en-us/azure/azure-monitor/reference/tables/azurediagnostics?source=recommendations#azure-diagnostics-mode-or-resource-specific-mode

This post discusses the following points:

5. How do we configure and test the new modernized resource-specific mode?

To test the resource-specific mode in this post, I am provisioning a random Recovery Services vault in a sandbox.

Step 1 – Go to Azure Portal > Recovery Services Vault > Create.

Here I create an Azure Recovery Services Vault Backup Policy in an effort to generate a database entry in the AddonAzureBackupPolicy table. The custom RSV Backup Policy is called “AllensWeeklyBackupPolicy” (which i will use below in my KQL query)

In the Azure Portal > Log Analytics Workspace > Logs > I  ran a query on the AddonAzureBackupPolicy database to confirm that a database entry was created in the resource-specific table.

AddonAzureBackupPolicy|
 order by TimeGenerated desc | where PolicyName contains "AllensWeeklyBackupPolicy"

The end-user verification is successfull since we can see that the resource-specific AddonAzureBackupPolicy database has been populated with the new Azure Backup Policy log showing the AllensWeeklyBackupPolicy provisioning.