As long as there has been business, there has been data. Unlike some of the newer technologies that have popped up in the past century, companies have always been gathering data. It may not have been the same type or amount that companies collect now, but it was always there.
But with the boom in big data, a new challenge has arisen. How do we keep track of all of this data? And perhaps even more important, how do we manage this data in such a way that we can gain useful insights from it?
“We’ve always had logs and we’ve always had to do something with them and try to analyze them and gain insights from them,” said Rich Lane, senior analyst at Forrester. “Now what we’re trying to do is connect the dots.”
This is where log management comes in. According to Nick Carstensen, technical product evangelist at log management provider Graylog, log management is a way of centrally managing and gathering all of the logs an organization is generating. Typically, this involves making them accessible in an easy and simple front end for viewing.
RELATED CONTENT:
How does Graylog help companies with log management efforts?
A guide to log management tools
Keeping logs is important because it gives you an idea of what is going on in your organization. A few things that a company can learn from logs are what time people are coming in in the morning, if they’re accessing files after hours, and if new software is being installed in the environment. “All of that can be detected and actually recorded through log management,” he said.
“Essentially, most of the monitoring platforms today want to take in log information, be able to put analytics across that, be that machine learning or other advanced algorithms and with the other data in the enterprise correlate it all together and give you better insights,” said Lane.
In the past, organizations found log management to be a hard problem to solve, Carstensen explained.
This was because there was no easy way to gather up the logs an organization generated in a way that was useful. “When logs were gathered, they were never used as there were too many to look through, and were mostly kept for audit reasons,” Carstensen said.
But with current log management tools, it’s possible to take in high volumes of logs and then index them for quick viewing and searching.
A properly configured log management solution can help ease the workload of IT staff. Most log management solutions will allow for alerting. This means that organizations can proactively monitor their environment. “These tools also allow for alerting on the logs to be more proactive than reactive, and even can give you trends over time to understand patterns in your organization. The ability to archive years of data and recall it when needed has made centralized log management more attractive than ever,” said Carstensen.
Log management solutions can allow IT teams to quickly pinpoint the location of an error or get alerted about upcoming issues so that they can respond faster. “When you do not have a central log management, logging into every device to go through the logs can take many hours, where a quick search could find the same data,” he said.
Typically, the responsibility of log management comes down to IT, but sometimes the responsibility can branch out to other groups. Depending on the size of the organization, sometimes log management can become something that the audit team would have a part in.
The audit team would typically only be responsible for the configuration of the tool, in terms of specifying the length of time logs are retained, what reports will be generated, and what alerts will be generated. The IT team would oversee the regular use of the tool. “Audit will normally ask for data from the log management system through the IT team, but will not manage or maintain the system on a day-to-day basis,” said Carstensen.
Carstensen explained that within IT teams, the owners of particular data will have the logs for that data. Networking teams will own the networking data logs, while a Windows infrastructure admin would own the data relevant to that. This can cause some issues in cases where individuals or teams don’t want to share data with each other.
This can be a major challenge when trying to implement log management solutions. In order to be successful, it’s important to break down those siloed groups so that the data can be centrally managed.
Having a top-down push for centralized log management can help break those silos. Carstensen explained that often when there is an issue in an organization, groups will pass it off to another team, saying it is not their issue. This results in long delays in actually fixing the issue. by allowing read-only access to all teams, teams will be able to quickly find the root of issues. This will enhance the stability and security of the organization as a whole.
According to Carstensen, another challenge associated with log management is getting proper buy in from upper management.
Another thing to keep in mind when looking into log management solution is making sure you have the proper storage. organizations need to keep in mind that big data takes a lot of space.
When looking for a log management solution, it’s important to carefully look at what features a vendor is offering.
Carstensen believes that the most important feature to consider is the ability to class logs from different sources. Windows generates logs differently than Linux, so if you’re running both operating systems in your environment, you want to make sure you can gather logs from each.
The second thing to keep in mind is the retention strategy. Organizations need to ask themselves how long they want to keep logs for. “Is it 30 days, a year, five years? You want to make sure the solution can support that ability to go longer,” Carstensen said.
The third consideration is ease of use. Carstensen explained that a lot of log management systems are powerful, but may take a lot of effort and staff to use. “You want something with minimal amounts of overhead to run with the flexibility to search and find your data very quickly.”
And as with any solution that an organization would be implementing, security is a concern that organizations will have to consider.
Logs contain information about authentication systems, what software versions are in use in your organization, and the types of security tools that are in place. “Having access to all that information will help the attacker know your network before the attack begins,” Carstensen said.
Carstensen recommends that organizations limit access to the log management tool in order to keep logs secure. He also recommends that the data be encrypted and an audit trail be kept in order to understand who is accessing the logs and what they did with those logs.
He also recommends organizations regularly evaluate access to the system. If during evaluations it is determined that access for certain individuals needs to be different, you can be a bit more proactive about it.
One trend that Carstensen has been seeing recently is a desire to enrich data. Log data itself has been pretty stagnant. Older router switches and firewalls could have been generating the same data for the past 20 to 30 years. Now the question is: “How do you enrich that data further by taking their IP address and then using a threat intelligence lookup feed or correlation engines to take that and say [let’s take out this IP that’s part of a known botnet attack] right now, or is this IP known to be malware? So we can enrich your data and make better alerts and decisions based on that.”
Carstensen predicts that log management will become easier to use going forward. With the addition of intelligence to data, it will be easier to create a baseline of trends and see when those change over time.
According to Forrester’s Lane, log management vendors moving forward will need to come up with better ways of storing this data. In particular, they need to move away from storing data in its raw format. Organizations need to be able to filter out the noise in order to get good insights, and a lot of log data is just repetitive information. Sort of the trend now that people are starting to try to figure out is how do we handle all that massive amounts of data without having to keep it around … I would say a huge proportion of logs themselves aren’t very meaningful. Right now, for most solutions, we have to take all that data in even if it’s something we don’t want or don’t need,” said Lane.
This guide is part of a larger Log Management Buyers Guide, which features additional articles and resources on log management. You can download a free copy here.
