Be Good to Your Future Self: Log the Right Stuff!

The next post will have code. I swear! But before we get to actual code, there’s one more foundation element I’d like to discuss: logging!

If humans have ever used your applications, you know that logs can help you diagnose problems. It’s pretty obvious that you can get your customers back up and running more quickly with the right log information. But those humble files have much more potential that just diagnostics!

Logs, Huh, What Are They Good For?

Plan what to log before you have to log it. Otherwise, you'll make the puppy sad. You don't want to make the puppy sad, do you? — Plan what to log before you have to log it. Otherwise, you’ll make the puppy sad. You don’t want to make the puppy sad, do you?

Logs come in a variety of formats, from the friendly and easy to mine text logs of Tomcat and Red Hat 6, to the binary event logs under Windows, to the binary and annoyingly hard to parse Linux/Systemd logs.* But no matter what their format, logs help us diagnose problems. A customer reports that trying to enter data into a certain form produced an error? A check of the Tomcat logs an tell us what the application thought went wrong. That’s assuming, of course, that we told the application to log the correct information.

That’s the key point: Logs are only as useful as we plan for them to be.

So to answer the question, “What are logs good for?” we have to ask another question: “What do you want them to be good for?”

* I’m partly joking. There’s a war within the Linux community between those who prefer the text logs produced by processes based on traditional UNIX principles and the binary logs produced by the monolithic Systemd process. You can tell which side has my sympathy by how I phrased that! I’ve never been a fan of a company forcing me to re-write many of my operational procedures without giving me any new functionality (for my business, anyway) in exchange — all the while making my life more difficult! Yes, I know that this or that person may benefit, but I didn’t, and I’m responsible for my company and its operations. So I’m not happy at the prospect of absorbing changes at a cost that I can’t recover.

Listening to Advice from the Wise Ones

The Wise Owl says it's always better to log stuff for law enforcement -- before they ask for it! — The Wise Owl says it’s always better to log stuff for law enforcement — before they ask for it!

In my post about legal compliance, I mentioned National Institute of Standards and Technology (NIST) Special Publication (SP) 800-53 (rev4). Is stakes out positions on just about every conceivable aspect of application security, including, of course, auditing and logging. Consider the control group for auditing called AU. It talks about generating audit logs and discusses some of the things a developer should consider. For instance:

AU-2 Audit Events, clause a: Determines that the information system is capable of auditing the following events… Where “following events” must be defined in your organization’s policies.

Why call this out? Because it’s foundational. Before you write a single line of code, you should know what kinds of events you want to track. Diagnostic information is a given. It’s easy to see how logging problem data can help you: using it, you can restore function to a site more quickly! But there are other things you should consider logging:

Authentication events: Take a step back from diagnosing problems for a moment. Let’s say a customer calls you to say that they think malware at their site managed to compromise their ID and password. They call you and ask you to give them a list of all the times they logged in between two dates and times. Can your logs provide that information?
Admin functions: What if that same user wants to know if their ID had been used to perform admin functions, like adding or deleting ID? Or changing passwords? Can you answer that question?
Forensic events: What if your business is compromised by an Advanced Persistent Threat (APT), and management calls the FBI for help. What if the FBI calls you and asks you for all activity for a certain ID between certain times? Or if they want to see the failed login attempts during a certain period of time? In a world where Russian hackers appear to be trying to affect the United States presidential election, this scenario is not all that far fetched. How disappointed would your management be in you if you can’t provide the information?

tl;dr version: Know your customers to anticipate what information they might need. Know your industry to know what kinds of forensics you might need. Finally, know your application to anticipate what kinds of diagnostics you might need to perform.

There’s another NIST 800-53 rev4 control that you might find interesting:

AU-12 Audit General, Control Enhancement 2: The information system produces a system-wide (logical or physical) audit trail composed of audit records in a standardized format.

That seems just reasonable. Why log something if you can’t read it later? But never underestimate the power of human creativity! I can’t count the number of times I’ve seen developers want to tweak standard log format for what seemed like valid reasons, but that ended up just being hard to read. The web comic xkcd addressed this. Not sure what standard to use? Investigate the defaults within your application servers or other environments. Tomcat does a good job of logging; so does Apache. Not only will the logs be easier to read, the chances of an off-the-shelf log analysis package being able to work with your logs is much greater if your log’s format is standard.

Implement the Change Where It Costs the Least

It's least expensive to decide what to log before coding. But it's never too late! — It’s least expensive to decide what to log before coding. But it’s never too late!

It’s easiest, and least expensive, to formulate your plan for auditing before you write any code. Working with all of the stakeholders (like the business owner, product owner, operations team, or really anyone with skin in the game), you can come up with heuristics defining what you’re going to log and how you’re going to log it. Collectively, that’s called your Logging and Auditing Policy. Pass that out to all software architects, designers, and developers so each of the applications you write can implement that policy. Everyone, from your developers to your support staff to your customers, will know what they can get out of the logs.

But even if you have an existing code base (and even if it’s huge), there’s still benefit to this exercise. It’s never too late to improve your code. It’s never too late to improve how your applications can support not only your customers, but your own company’s legal interests.

It’s hard to argue with a more successful career and less imperial entanglement!

Have you had any experience establishing an Audit/Logging Policy? What did you find was helpful and effective?

by Terrance A. Crow

Terrance has been writing professionally since the late 1990s — yes, he’s been writing since the last century! Though he started writing about programming techniques and security for Lotus Notes Domino, he went on to write about Microsoft technologies like SQL Server, ActiveX Data Objects, and C#. He now focuses on application security for professional developers because… Well, you’ve watched the news. You know why!