Introduction

I've been bouncing around logging software for awhile, and decided to give Splunk a try (over Kibana) so this is just a quick little tutorial on how to set it up.

Prerequisites

• Splunk Service with IP (Cloud or Local or Docker works)
  • Mine is a docker container with the IP 192.168.68.46:8000
• Docker container that you want to have logged into Splunk

What is Splunk?

Splunk is used for monitoring and searching through big data. It indexes and correlates information in a container that makes it searchable, and makes it possible to generate alerts, reports and visualizations.
Cloudian

Essentially it is a place that collects and gathers all your logs from different locations which allows you to consolidate them and search them for specific information

Target Docker Container

This is out of the scope of the project, but I have a simplifed code that runs a .NET application which Hangfire to run scheduled tasks at a specific interval. The contents of the docker container and what it does really doesn't matter at all, as long as it logs and follows some sort of format that you can parse. I will be following the container below for C# which utilizes Nlog to format all of my output into a consistent format.

using NLog.Config;
using NLog.Targets;

namespace Application.Common.Utilities
{
    public class ApplicationLogging
    {
        public static void Initialize()
        {
            var config = new LoggingConfiguration();
            var consoleTarget = new ConsoleTarget
            {
                Name = "console",
                Layout = "${date}|${level:uppercase=true}|${callsite:includeNamespace=false:className=true:fileName=false:includeSourcePath=false:methodName=false}|${message}",
            };
            config.AddTarget(consoleTarget);
            config.AddRuleForAllLevels(consoleTarget);
            LogManager.Configuration = config;
            LogManager.GetCurrentClassLogger().Info("Initialized my Logging!");
        }
    }
}

The above code is just the initialization for NLog library in .NET. You can use any logging application, or even native console.log to output it in a specific standardized format. The above code generates a log that looks like

2021/11/01 19:31:31.768|INFO|ApplicationLogging|Initialized my Logging!

This follows the format of
<TIME>|<LOG_LEVEL>|<CALLING_CLASS>|<MESSAGE>

• <TIME>- Time that the log is generated by the application, NOT the time it was received by splunk (though the same)
• <LOG_LEVEL> - The severity of the log, DEBUG,INFO,ERROR,etc. You can sort in splunk on this field easily
• <CALLING_CLASS> - In C#, I value knowing what the exact class is writing to the logger when debugging
• \MESSAGE> - Whatever other information that you want to pass to the log.

Splunk Instance (Cloud, Local, or LAN)

For this it will work with any type of Splunk so you can host it anywhere. For me I have it as a docker container on my network that all my other containers are able to see and thus write to it.

1. Navigate to your Splunk Web UI and log in at http://192.168.68.46:8000 (Port 8000 is default web port)
2. Navigate in the top menu bar to Settings > DATA > Data Input
3. Navigate to HTTP Event Collector, Recieve data over HTTP or HTTPs
   
4. Navigate to New Token in the top right corner of the page
   
5. Add a name and description for your collector, Click Next
   
6. Click through Next for Input Settings, Review, and Done. Default settings are fine
7. Save the Token Value in this final generation page, you will need that
   
8. Navigate to your docker container page and add the following arguments to your docker container
   --log-driver=splunk
--log-opt splunk-token=7fc97e9f-e1o6-4tr2-8d8a-0b51d11f83bd
--log-opt splunk-url=http://192.168.68.46:8088
   • --log-driver=splunk - This tells the docker container to use a different logging driver
   • --log-opt splunk-token={TOKEN} - This is the Splunk token you got from HEC creation page
   • --log-opt splunk-url - The endpoint for your splunk instance, port defaults to 8088 (different from web ui)
9. Restart your container and log into Splunk and search at http://192.168.68.46:8000/en-US/app/search/search
   source="http:siglerdev-blog" | rex "\{\"line\":\"(?<time>.+)\|(?<log_level>.+)\|(?<Controller>.+)\|(?<Message>.+\}|.+?)\"" | fields - _time | table time log_level Controller Message | sort -time
   

Once you've reached this far you can experiment with the different parameters for the query and pull different information. You can log it in a bunch of different formats but I really like my format since it has all the information that I need.

Thank you for reading and let me know below if you have any questions!