Leveraging Your Log Data with Azure Log Analytics

With Azure Log Analytics, you can gather and search log data from all of your resources, whether they reside on Azure, on another cloud host, or on-premises. Then you can transform the gathered data into rich analytics with AI-enhanced insights into your environment. To get started, choose between the free and paid options, then follow the straightforward steps to create a workspace, configure your data sources and add solutions. Once your resources are fully connected and configured, it’s time to start using log searches to query your data. Log Analytics lets you save and automate your searches, then add them as visualizations to your dashboard. You can also export log search results, or access them with custom apps through the log search API.

In the wake of a major update in October, this is an ideal time to try Log Analytics, which now benefits from a new query language and a wealth of new capabilities that support practically limitless data exploration.

Options and pricing

Before getting started with Log Analytics, it’s a good idea to become acquainted with any costs you may incur as you use the service. Log Analytics changed its pricing structure in October 2016. Any accounts created since that time will fall under the new pricing structure, which comes in two tiers: free and paid. The free tier has a daily data quota of 500 MB, and a data retention period of 7 days, without an option for elongated retention. The paid tier, on the other hand, has no daily data quota, and comes with a default retention of one month, which you can elongate, subject to relatively modest overage charges. The price for the paid tier is $2.30 per GB of data uploaded to Log Analytics; and the overage charge for data retained beyond one month is 10 cents per GB per month.

Creating an OMS workspace

As part of the Log Analytics sign-up process, you will need to create an OMS workspace that will house your collected data, your data sources, and any management solutions you choose to add. If you wish, you can create several OMS workspaces to cover several separate environments.

Collecting log data

Log Analytics can gather data from connected resources that are either internal or external to Azure. For internal, Azure resources, Log Analytics gathers data via Azure Monitor and Azure Diagnostics. If you want to collect data from one or more of your Azure resources, and you haven’t already setup Azure Monitor, then the Azure team recommends that you do so as a prerequisite for leveraging Log Analytics. For connected resources outside Azure, Log Analytics will draw data generated from agents on Linux or Windows machines. These agents can either be connected directly to Log Analytics, or through a System Center Operations Manager management group.

There are two ways to connect your resources: configuring data sources, and adding management solutions.

Configuring data sources

A data source is a specific kind of data. A single connected resource can generate data from several different data sources. Common data sources include the following:

• Events
• Performance data
• IIS logs
• Custom text logs

Although the record types corresponding to different data sources and solutions will differ, Log Analytics makes it possible to query and analyze the various record types collectively. However, you will need to configure the data sources corresponding to each type of log data that you want to gather. Then the Log Analytics service will take care of communicating the configuration to each connected source.

Adding management solutions

If you need to generate data that addresses a specific problem area, Log Analytics management solutions can provide a pathway. You can browse and add management solutions through either the Azure Marketplace or the management solutions gallery. For an example of how management solutions work, have a look at the widely used Change Tracking solution for tracking and visualizing changes that take place to your installed software.

Data collected through a management solution will be sent to the OMS service, where it will generally appear within an hour. Note that each management solution will add to the total quantity of data used. If you are using the free pricing tier, then this will contribute to your daily data quota. Therefore, free tier users should remove management solutions when no longer using their data.

Log searches

Log Analytics enables you to create sophisticated log searches of your data. After saving a log search, you can repeat it anytime, or you can set the search to run automatically.

Alerts

You can easily arrange the system to alert you when an automated log search returns results that meet your criteria. For example, you can setup alerts to notify you when automated log search results return values within in a certain range that may indicate a problem in your environment.

Visualizations

You can also create rich visualizations to help you analyze data gathered from specific log searches. After creating a log-search visualization, you can then add it to your analytics dashboard for at-a-glance viewing at any time.

Exporting log data

Log Analytics has made it easy to export log search results for use in Excel, Power BI and other apps.

Log search API

Log Analytics provides a log search API so that you can efficiently build custom apps that pull data from the Log Analytics service.

A new query language

The Azure team recently announced a new query language for Log Analytics. When contrasted with the service’s legacy language, the new language holds several advantages:

• A syntax that is closer to SQL and natural language, making it easier to learn and use.
• Extensive piping capabilities, facilitating more advanced, complex queries.
• More sophisticated date/time functions.
• More advanced join capabilities, such as support for inner and outer joins.
• More advanced calculated fields for additional commands. 
• An Advanced Analytics portal with multiline query editing, as well as new visualizations and diagnostic tools.
• Smart Analytics for recognizing meaningful patterns and comparing separate datasets.

Other benefits of the update include improved Power BI integration, greater consistency with other Azure services, and a number of other significant improvements. For an in-depth discussion and demonstration of the new query language, see Azure’s video tutorial.

Leveraging the new Log Analytics

As outlined above, whether your resources reside on Azure, on another cloud host, or on-premises, you can now collect, search and analyze their log data with Azure Log Analytics. The service offers two pricing tiers: free and paid. If you choose the free tier, then you can collect up to 500 GB of data per day and keep it for 7 days. If you choose the paid tier, then you can collect unlimited data at $2.30 per GB and keep it for one month (at no additional cost) or for an extended period (for a modest fee). To get started with Log Analytics, you’ll need to create an OMS workspace to serve as your data repository. Next, you’ll need to configure your data sources and add management solutions, such as Change Tracking, so that you can start efficiently collecting log data. With your resources connected, your data sources configured, and your management solutions in place, it’s time to start querying your data with log searches, which you can save and automate. You can also add log search visualizations to your dashboard; export log search results for viewing in other apps; or access log search results with your own, custom apps through the log search API. On the heels of a revamp of the log search query language, it’s an ideal time to try Log Analytics, which now benefits from a more intuitive syntax and a long list of new, advanced features that will fuel ever-deeper insights into your data.

For more information on Azure Log Analytics and other Azure services, contact us.