Thursday, February 23, 2017

Setting up a Pentesting... I mean, a Threat Hunting Lab - Part 6

I wish I had an EDR vendor send me a dev agent [hint! hint!] to test how much event data I can capture from an endpoint, but for now I love to use Sysmon when it comes down to endpoint visibility. In this post I will show you how to install sysmon and use custom configurations to filter noise and still get the visibility you need to hunt for advanced adversaries. Additionally, as you might already know, we need some type of log forwarder to send logs to our ELK stack. In case you didn't know, Elastic provides several products besides Elasticsearch, Logstash and Kibana, and the one that will help us live stream Windows event logs to our ELK stack is named WinlogbeatIn this post, I will also show you how to set it up and integrate it with our ELK stack configurations.



Getting Started with Sysmon

Sysmon Overview

Sysmon is a Windows system service and device driver that, once installed on a system, remains resident across system reboots to monitor and log system activity to the Windows event log. In contrast to common Antivirus/HIDS solutions, Sysmon performs system activity deep monitoring, and log high-confidence indicators of advanced attacks. Sysmon is using a device driver and a service that is running in the background and loads very early in the boot process. [Source]

Sysmon Tags and Events




Process Create
File creation time
Network connection detected
Sysmon service state change (cannot be filtered)
Process terminated
Driver Loaded
Image loaded
CreateRemoteThread detected
RawAccessRead detected
Process accessed
File created
Registry object added or deleted
Registry value set
Registry object renamed
File stream created
Sysmon configuration change (cannot be filtered)
Named pipe created
Named pipe connected

Sysmon XML Conditions



Default, values are equals
is not
Values are different
The field contains this value
The field does not contain this value
begin with
The field begins with this value
end with
The field ends with this value
less than
Lexicographical comparison is less than zero
more than
Lexicographical comparison is more than zero
Match an image path (full path or only image name). For example: lsass.exe will match c:\windows\system32\lsass.exe

Great Sysmon Use Cases

  • BotConf2016 Advanced Incident Detection and Threat Hunting using Sysmon (and Splunk) Tom Ueltschi 2016 (Slides)
  • BotConf2016 Advanced Incident Detection and Threat Hunting using Sysmon (and Splunk) Tom Ueltschi 2016 (Video)
  • Microsoft Sysmon Deployment 2017 - Dimitris Margaritis (slides)
  • Splunkmon - Takin Sysmon to the Next Level (Whitepaper)
  • Posh-Sysmon Module for Creating Sysmon Configuration Files 2017(Article)
  • How to Go from Responding to Hunting with Sysinternals Sysmon RSAC 2017 (Slides)
  • Hunting with Sysmon - Michael Haag 2017 (Article)
  • Sysmon-dfir - Michael Haggis (Github)
  • Sysinternals Sysmon unleashed (Article)
  • SwiftOnSecurity - Sysmon Config 2017 (Github)
  • Explaining and adapting Tay's Sysmon Configuration (Article)

Installing Sysmon

I had already installed Sysmon V5 in my systems, but with a recent update from Mark Russinovich, I needed to update a few images, and content in this post. 

Figure 1. Sysmon V6 FYI release.

To get started download Sysmon V6 from here

Figure 2. Sysmon V6 Download page.

Extract contents of the zipped file to a preferred directory.

Figure 3. Extracting files to tools directory. 

Launch cmd.exe as administrator, navigate to the folder where sysmon was extracted to, and if you want to know what sysmon can do just type:

sysmon.exe /?

Figure 4. Sysmon Menu.

We can go ahead and try a basic installation by running the following command:

sysmon.exe -i -accepteula -h md5,sha256,imphash -l -n

  • - i : Install Service and driver. Optionally take a configuration file.
  • -h : Specify the hash algorithms used for image identifications
  • -l  : Log loading of modules. Optionally take a list of processes to track
  • -n : Log network connections

Figure 5. Installing Sysmon.

Now, if we run Eventviewer as administrator and browse to Applications and Services Logs > Microsoft > Windows > Sysmon > Operational, you will see that Sysmon is already working and generating logs as shown in figure 6 below.

Figure 6. Sysmon Logs. 

You can also view the configuration which Sysmon is running on by typing:

sysmon.exe -c 

Figure 7. Current Configuration. 

How do we update our current configuration and apply rules to it?

sysmon.exe -c Your_custom_config.xml

You can use my StartLogging.xml config as a basic first script to start. This script has several Event IDs set to Log everything, and this is because I want you to tune it your way.. I just had a few exclusions already set there specially for Event IDs 1, 3, 6, 7, 10, 11, 12,13,14 to help you a little bit with filtering some initial noise.

Lets update our current config and apply our StartLogging.xml configuration. 

Figure 8. Updating current configuration and showing it on console.

You will not see many logs being generated, but that will change as soon as you start testing a few things. We have not set up our winlogbeat data shipper, but I will highly recommend to turn winlogbeat services off until you tune your sysmon configuration so that it captures the main anomalies from the attack you are executing. Once you are comfortable with your sysmon config, turn on your winlogbeat service and you will be able to see the events in your Kibana dashboard.

This can be a good time to take a snapshot on your endpoints with Sysmon installed.

Getting started with Winlogbeat

Stay in the windows computer where you set up Sysmon. To get started, download Winlogbeat from here and copy the unzipped folder to C:\Program Files\ as indicated by the Getting Started Winlogbeat Guide.

Figure 9. Winlogbeat Download page.

Figure 10. Save the zipped file to your drive.

Figure 11. Winlogbeat folder unzipped.

Figure 12. Copying winlogbeat folder to indicated location. 

Figure 13. Winlogbeat folder copied to C:\Program Files\ .

Open PowerShell as administrator, navigate to the winlogbeat folder contents and run the install-winlogbeat-service.ps1 powershell script.


Figure 14. Run PowerShell as administrator.

Figure 15. Navigate to your winlogbeats folder.

Figure 16. Running install-service-winlogbeat.ps1 script. Also, using Run Once option.

Next, run notepad as administrator. This will allow us to edit the winlogbeat config file and saving it without having "access denied" warning messages. Once notepad is open, open the winlogbeat.yml file in your winlogbeat folder.

Figure 17. Opening notepad as administrator.

Figure 18. Opening winlogbeat.yml file in notepad.

You will now be able to edit what log types it collects. We will add the following line after - name: System

Add:  - name: Microsoft-windows-sysmon/operational

Figure 19. Original first part of the winlogbeat config.

Figure 20. Adding Sysmon Logs to the configuration. 

Next, you can see that elasticsearch by default is configured to be the output to use when sending the data collected by the beat. We are going to change that by doing the following:

Add a # sign before Output.elasticsearch (as shown in figure 22)
Add a # sign before hosts: ["localhost:9200"] (as shown in figure 22 below)

Figure 21. Original Elasticsearch output configuration.

Figure 22. Adding # signs to the Elasticsearch output section to disable the output.

Now, it is time to configure the Logstash Output of the winlogbeat configuration. As you can see, we can set up the certificate that we created in our Ubuntu server in our previous post. We are going to make some changes to this part, but first make sure  you have your cert created as shown in figure 24 below. If you did not create the certificate and private key in the previous host, I recommend to go back to it and do it before we continue.

Figure 23. Original Logstash Output section. 

Figure 24.  Making sure cert exists in our ELK server. 

Use PSCP.exe to retrieve the certificate from the ELK server. Make sure SSH services are running in your ELK server before doing this. You can download PSCP.exe as part of a Putty bundle from here.

Figure 25. PSCP.exe to retrieve certificate from ELK server via SSH.

Once you are ready run the following:

.\PSCP.EXE <username>@<IP of you ELK>:/etc/pki/tls/certs/logstash-forwarder.crt C:\<anywhere in your windows system>

Figure 26. Copying certificate from ELK to your Windows computer. 

Figure 27. Certificate was copied successfully. 

Now, back in our Logstash output section, do the following:

  • Delete the # signs from Output.logstash (as shown in figure 28)
  • Delete the # signs from hosts: ["Your ELK IP address:5044"] (as shown in figure 28)
  • Delete the # signs from ssl.certificate_authorities: ['C:\<wherever you copied your cert to']  (as shown in figure 28)

Figure 28. Logstash configured to be the default output.

Test your winlogbeat config by running:

.\winlogbeat.exe -c .\winlogbeat.yml -configtest -e

Figure 29. Config OK.

Finally, you will have to start the winlogbeat service. You can use your PowerShell to do it by typing:

start-service winlogbeat

Figure 30. Starting the winlogbeat service.

Figure 31. Starting the winlogbeat service.

Figure 32. The winlogbeat service started successfully. .

Next, open your favorite browser and go to your ELK's IP address. You will be able to substitute logstatsh-* for winlogbeat-* and set the field name to Timestamp as shown in figure 33 below. Click on the create option at the bottom to continue. 

Figure 33. Index Pattern configuration.

Figure 34. Index Pattern configuration.

You will be prompt to see how your winlogbeat-* index is storing the logs hitting your ELK. Next, on the left panel options, click on Discover to look at your logs. 

Figure 35. Winlogbeat index.

Figure 36. Searching for logs.

Figure 37. Logs made it to our ELK stack.

Below your Winlogbeat-* index, you can actually hover the data fields and click on the option add. You will be able to add as many fields as you want making it easier to read the logs.

Figure 38. Adding fields to our discover view. 

WOW! We made it! We have officially finished setting up our basic threat hunting lab. Remember, this is just the beginning. There are so many things that you can add to this lab setup, but I will leave that to your imagination and your own research. 

Now, the one million dollar question,


I will give you a sneak peek of what I will do in my next series "Chronicles of a Threat Hunter" with a basic example:

Hunting for users in the network interrogating AD to know who the Domain Admins are

Figure 39. Net.exe interrogating our AD for Domain Admin users.


(event_data.ObjectType:"SAM_GROUP" AND event_data:ProcessName:"C:\\Windows\\System32\\lsass.exe") OR event_data.CommandLine:"Domain Admins"

Figure 40. Hunting for the HUNTER :) . 

That's it, and I hope you found this series helpful to start setting up your own lab. Let me know how this works out for you. #SharingIsCaring


Feedback is greatly appreciated!  Thank you.


  1. Thanks a ton for this awesome post.

    Please fix:
    start-server winlogbeat => start-service winlogbeat

    Although i had few problems initially mostly due to connectivity between the windows and ubuntu vm. Firewall on ubuntu was blocking connection from windows vm :).

    Still i am facing an issue where using the certificates for ssl connection prevents sending winevents back to logstash (ubuntu vm).

    But finally got it all setup without ssl part in the configuration as of now.
    Based on a hint from this reference:

    # The Logstash hosts
    hosts: [""]

    # Optional SSL. By default is off.
    # List of root certificates for HTTPS server verifications
    #ssl.certificate_authorities: ['C:\elk\certs\logstash-forwarder.crt']

    Quite a fun !!

    Thanks again.

    1. Hey sessionpool, Thank you for the feedback. I wonder if there is something missing from the Logstash configuration from starting from Figure 37. Specially in figures 40 & 41. Make sure the SubjectAltName is set up and pointing to your Host's IP before you create your certificate. Let me know if this helps. Thank you again for your feedback and glad you are having fun with these tutorials :)

    2. I got my already programmed and blanked ATM card to withdraw the maximum of $1,000 daily for a maximum of 20 days. I am so happy about this because i got mine last week and I have used it to get $20,000. Mike Fisher Hackers is giving out the card just to help the poor and needy though it is illegal but it is something nice and he is not like other scam pretending to have the blank ATM cards. And no one gets caught when using the card. get yours from Mike Fisher Hackers today! *email

  2. there's some typo in the query field. i think should be:
    (event_data.ObjectType:"SAM_GROUP" AND event_data.ProcessName:"C:\\Windows\\Sytem32\\lsass.exe") OR event_data.CommandLine:"Domain Admins"

    BTW, very good tutorials and step by step configuration. Thank you a lot!

    1. Hey teedeearr, thank you for the observation. Fixing it now. I think it was really late when I typed that query :) . I am glad to hear you found the tutorials very helpful :)

  3. Thanks for this Wardog, any idea how you can deploy sysmon on +3K workstations ? Without downtown ?

  4. Thanks for the great writeup! Why would you exclude Lsass from ProcessAccess? Also is there a parameter option that i could pass to Sysmon to enable ProcessAccess instead of a config file? Thanks.

    1. Hey Abdulellah, I am excluding Lsass from ProcessAccess but as a the source and not the target. There are several processes that Lsass usually interacts with and causes several FPs. I was just excluding the processes that were very noisy as "source". I like to monitor lsass as a target for any suspicious access from a suspicious process. I can enable lsass as a source in the config but it will not tell me much. I feel that lsass as a target could tell me more and would be more interesting to investigate (just my opinion). Regarding enabling ProcessAccess w/o a config file, I dont think it is possible. The only arguments accepted via the sysmon binary are just to enable modules being loaded, hashing, and network events. The rest needs to be specified via a config file. I hope this helps.

  5. Hello Wardog. Thanks for the fantastic write up. I set up the environment flawlessly. Looking forward to more of your content. It's the best I've found on log analysis with elasticstack, period. How can I stay breast of your future projects? Thanks.

    1. Hey Z3nyth! sorry for the late response, I really appreciate the feedback and I am glad that you were able to install your environment without a problem :) :) I am in the process of adding more things to the HELK build and you can check all the updates by following me in Github or twitter:
      Twitter: @Cyb3rWard0g

      I hope you have a great weekend!

  6. Hi, I followed everything step by step however I am stuck on figure 33/34... First of all, I don't have an option to tick "Index contains time-based events" - this does not exist. Secondly, nothing is happening when I type Winlogbeat-*. I still unable to fetch mapping and cannot select any Time Filter field name. I am not sure what is going on as I followed everything step by step and everything up to figure 33 is working fine exactly as described in your tutorial. Any ideas please?

    1. Hey Nleocenlony! mhhhmmm what version are you running and is you endpoint sending logs to your ELK? it seems that you have the ELK ready to receive logs but not creating the winlogbeat-* index. Have you check if your endpoint is sending data?? can you disable the local firewall?? try to ping the ELK ip and see if you get a response. Also, what I would do is check your logstash logs : tail -f /var/log/logstash/logs* and see if there is anything stopping you from getting logs. Let me know what happens after checking all that. Happy to help !

    2. Thanks for the reply. I am not sure how I've done it but I finally managed to create winlogbeat-* index. Logs are now being sent and displayed in Kibana right away however, these logs are huge! And pretty useless obviously... I configured the template according to the guide but my log file is growing a lot every second. From 25 mb to 117 mb within a few minutes (from single end point) Please see this screenshoot -

      There are mainly thousands entries related to C:\Windows\system32\wbem\wmiprvse.exe. I am not sure if I configured something in a wrong way. Please see the screenshoot of my Kibana -

      When I checked logstash logs (tail -f) I've seen there are some entries which are similar to what Kibana is displaying:

      charlie@ubuntu:/var/log/logstash$ tail -f /var/log/logstash/logs*
      UtcTime: 2018-01-06 20:22:44.291
      SourceProcessGUID: {25851C25-60F3-5A51-0000-0010B05C0300}
      SourceProcessId: 2308
      SourceThreadId: 3808
      SourceImage: C:\Windows\system32\wbem\wmiprvse.exe
      TargetProcessGUID: {25851C25-60F1-5A51-0000-0010C32B0300}
      TargetProcessId: 2152
      TargetImage: C:\Windows\system32\svchost.exe
      GrantedAccess: 0x1410
      CallTrace: C:\Windows\SYSTEM32\ntdll.dll+5157a|C:\Windows\system32\KERNELBASE.dll+d817|C:\Windows\system32\wbem\cimwin32.dll+2a6c|C:\Windows\system32\wbem\cimwin32.dll+2291|C:\Windows\system32\framedynos.dll+84e0|C:\Windows\system32\framedynos.dll+914e|C:\Windows\system32\wbem\wmiprvse.exe+5f51|C:\Windows\system32\wbem\wmiprvse.exe+5cf0|C:\Windows\system32\RPCRT4.dll+323d5|C:\Windows\system32\RPCRT4.dll+269b2|C:\Windows\system32\ole32.dll+16f16e|C:\Windows\system32\wbem\FastProx.dll+d36d|C:\Windows\system32\ole32.dll+170ccd|C:\Windows\system32\ole32.dll+170c43|C:\Windows\system32\ole32.dll+2a4f0|C:\Windows\system32\ole32.dll+1714d6|C:\Windows\system32\ole32.dll+17122b|C:\Windows\system32\ole32.dll+16fd6d|C:\Windows\system32\RPCRT4.dll+250f4|C:\Windows\system32\RPCRT4.dll+24f56|C:\Windows\system32\RPCRT4.dll+2775b|C:\Windows\system32\RPCRT4.dll+2769b|C:\Windows\system32\RPCRT4.dll+27632|C:\Windows\system32\RPCRT4.dll+2532d], :response=>{"index"=>{"_index"=>"winlogbeat-2018.01.06", "_type"=>"doc", "_id"=>"AWDNI9xfO1U7sD6nOi79", "status"=>404, "error"=>{"type"=>"index_not_found_exception", "reason"=>"no such index", "index_uuid"=>"_na_", "index"=>"winlogbeat-2018.01.06"}}}}

      Am I doing something wrong? I don't want to log unnecessary events like this. By the time I finish writing this post it generated THOUSANDS of entries lol. Thank you.

    3. Hey Nieoceniony! If you feel that the the logs are useless because they are too big (showing unnecessary information) or there too many of the same kind and they are useless, you can do two things. First, create a logstash filter to strip fields content or remove fields to avoid having information that will not be useful in an engagement. Second, update the sysmon config that I provided in this post. I just provided a basic initial config that could help understand what you can log with Sysmon. Do those two things and see if all that helps your use case. I like to have as much data as possible first, and then I start filtering my logs depending on my use case. If this is for production, then you will have to make decisions on what you believe will reduce your false positive rates. I hope this helps. You are not doing anything wrong. This is the beginning of your own ELK build. Now you can make it your own and start creating your own configs. I am in the process of creating a DocckerFile for my own ELK: . That one will expedite building one and will have more advanced analytic capabilities to play with the data. There I have examples of Logstash filters and index templates. Dashboards are already created (default ones) and other enrichments are available.

  7. really cool post, highly informative and professionally written and I am glad to be a visitor of this perfect blog, thank you for this rare info!
    splunk training in hyderabad

    1. Thank you very much for the feedback nasreen!! It means a lot :) I am glad it is helpful. Let me know if there is anything that I should add or change or talk about more :) I hope you have a great weekend!

  8. This comment has been removed by a blog administrator.

  9. This comment has been removed by a blog administrator.

  10. Hi Cyberwardog, this post is super useful, I am excited to see the logs come in ELK and be able to run queries and start hunting. The setup worked as expected on Ubuntu 16.04 LTS assessing the ELK server through IP address. Could you please show how to configure it when accessing through DNS name instead of IP address? Thank you.

  11. Hey, very nice site. I came across this on Google, and I am stoked that I did. I will definitely be coming back here more often. Wish I could add to the conversation and bring a bit more to the table, but am just taking in as much info as I can at the moment. Thanks .

  12. Thanks for posting useful information.You have provided an nice article, Thank you very much for this one. And i hope this will be useful for many people.. and i am waiting for your next post keep on updating these kinds of knowledgeable things...Really it was an awesome article...very interesting to read..please sharing like this information.

    Cpa offers

  13. Thanks for provide great informatic and looking beautiful blog, really nice required information & the things i never imagined and i would request, wright more blog and blog post like that for us. Thanks you once agianMarriage certificate in delhi
    Marriage certificate in ghaziabad
    Marriage registration in gurgaon
    Marriage registration in noida
    special marriage act
    Marriage certificate online
    Marriage certificate in mumbai
    Marriage certificate in faridabad
    Marriage certificate in bangalore
    Marriage certificate in hyderabad thanks once again to all.

  14. Thanks for provide great informatic and looking beautiful blog, really nice required information & the things i never imagined and i would request, wright more blog and blog post like that for us. Thanks you once agian

    special marriage act
    name add in birth certificate
    passport agent
    court marriage in delhi
    name change
    marriage registration
    birth certificate in gurgaon
    birth certificate in noida
    birth certificate in ghaziabad
    birth certificate in delhi

  15. Thank you so much for sharing this amazing blog, visit OGEN Infosystem for Website Designing Company in Delhi, India and also for SEO Service in Delhi.
    Web Development Company

  16. I am Happy with your website. I got Lots of Knowledge in your site. I have shared your site to all. I have some site for all. Thank you.

    Avast Login
    bullguard login

  17. Thank you for sharing this genuine blogspot with us. I like your post and now I am gone share it to my profile of facebook. Garmin Express by clicking on this link get know about me more.

  18. Thank you so much for this amazing information sharing with us. Visit Appslure WebSolution for the best mobile app development company in Gurgaon
    Mobile app development company in gurgaon

  19. This is realy a Nice blog post read on of my blogs It is really helpful article please read it too my blog Change profile picture on Spotify. you can visits our websites or toll free no +1-866-558-4555. solve your problem fastly.

  20. On the topic of endpoint testing, have you tried google rapid response (GRR)?

    This series was extremely helpful to me in setting up a lab, and I would love to see new options/utilities

  21. Hello Roberto,
    Its really a great write-up. But, I am struck at "Pulling from cyb3rward0g/helk-zookeeper" stage. Nothing is happening from there after.
    Kindly help.

  22. bullguard login- PC security is the way toward anticipating and distinguishing unapproved utilization of your PC. Counteractive action estimates enable you to prevent unapproved clients from getting to any piece of your PC framework. Recognition causes you to decide if somebody endeavored to break into your framework, in the event that they were effective, and what they may have done. | is-install-mdl-install | bullguard login

  23. McAfee activation www mcafee com activate is the essential process if you have McAfee antivirus on your device. To activate the McAfee anti-virus, you need to purchase the McAfee product key. This key can be obtained from activate card or online. | total protection

  24. Thanks for provide great informatic and looking beautiful blog, really nice required information & the things i never imagined and i would request, wright more blog and blog post like that for us. Thanks you once agian

    court marriage in delhi ncr
    court marriage in delhi
    court marriage in noida
    court marriage in ghaziabad
    court marriage in gurgaon
    court marriage in faridabad
    court marriage in greater noida
    name change online
    court marriage in chandigarh
    court marriage in bangalore

  25. The TomTom is the most feature-packed, best connected, and smartest navigation device on the market, packing a number of features which help it compete with your smartphone. | tomtom get started | | tomtom update

  26. Hii! I am Emma ava, I am working as a technician. If you are facing any kind of issues with aol email not working then just feel free to contact us at our technician team.

    aol mail not working on iphone

  27. If you have any problem related to Outlook, AOL, or Yahoo mail then contact experts who are experienced and have been trained to resolve any issues regarding Emails. You can also call us on our Email helpline number and get instant help.

  28. Looking for printer tech support then you're in the right place. You should talk to our technical support team to solve your printer related issues. Call Now: USA/Canada: (+1) 8884800288 & UK: + (44) 800 041-8324.
    Brother Printer UK

  29. Thanks for this informative information....If you are looking for how to resolve mcafee error 76567? No need to take worry, you can consult with our experts at USA/Canada: +1-855-869-7373 and UK/London: +44-800-041-8324. We are here to solve your antivirus issues.
    How to Resolve McAfee Error 76567

  30. Very informative post! This post gives truly quality information. I find that this post is really amazing. Thank you for this brief explanation and very nice information.

  31. To have cyber security skills you need to undertake cyber security training. This is a training that is offered in a number of government and private institutions. cyber security institute in hyderabad