RSA Netwitness Investigator Regular Expressions

In the blog post below (here) I talked about my theory to detect DGAs by looking for consecutive consonants in a row within a URL.  However my SNORT rule does not work like I wanted it to.  I thought the rule would look for the HOST: portion of the HTTP header and start the regex match after colon and whitespace. When what it seems to do is trying to match the whole line as one entity including the word HOST, this causes the match to fail if there are ANY vowels in the line vs. only caring if there are x number of consonants in a row before or after a vowel.  So I figured I’d leverage another tool, RSA’s Netwitness network forensics product would work. You can download a home version of the tool here.

Netwitness investigator allows users to use RegEx to filter the packet capture data including of course web surfing.  As I understand it they use the BOOST RegEx engine as opposed to the PCRE engine I’m more familiar with. Either way, there doesn’t seem to be a lot of documentation included with the product around the needed syntax. So I recommend you try just throwing in your regex and see what happens as Investigator will do some sanity checking. Taking my own advice I entered my SNORT regex as a custom drill in the alias.host field (a helpful message from an experienced Netwitness user made me realize I shouldn’t assume readers knew I was using a custom drill for this filter).  As you will quickly learn you need to backslash escape most non alpha-numeric characters until it works.  The pic below shows the error checking where I intentionally left off the backslash escape character in front of the comma in the repetition operator {9,}

regex_validation

I ended up with this which at least didn’t error out.

alias.host regex [-tnrshdlfcmgpwbvkxjyqz0-9]{9,}

You also have to be considerate of search time depending of course on how much data you’ve stored or how big your pcap is.  The above was taking a bit longer than I wanted so I figured I’d try to narrow down the search first by only applying the regex to traffic classified as http. So this was the next attempt

service = 80 && alias.host regex [-tnrshdlfcmgpwbvkxjyqz0-9]{9,}

even though Netwitness designates the service as 80 it is not looking at the port to decide on whether the traffic is http.  So within your results you will see anything detected as http despite the port number.  However, this left me with the false positives I talked about in my last article where the left most or host portion of the URL is much more frequently randomized characters then I would have thought, mainly related to cloud computing/content delivery networks. Example pic

Initial RegEx Match

initial_regex_matches

So yeah that’s a bit noisy but will detect the most randomized domains.  I wanted to see if there was a way to limit the cloud provider’s silliness.  The problem is don’t want to anchor it to end of string because then I have to assume the TLD is of a certain length.  I need to match randomstuff.com, randomstuff.co.cc, and randomstuff.hinet.cn so I figured I’d just match a literal period before the consecutive consonants. To do this you need to double escape the periods to make it a literal match.

So that leaves me with two different searches one broad one narrow.

Broad
service = 80 && alias.host regex [-tnrshdlfcmgpwbvkxjyqz0-9]{9,}

#As mentioned before it will look for x number of consonants, numbers, or hyphens in a row anywhere within the domain requested with HTTP protocol.

Narrow
service = 80 && alias.host regex \.[a-z0-9-]*?[-tnrshdlfcmgpwbvkxjyqz0-9]{9,}

#This more restrictive version will look for a literal period then optionally zero or more letters, numbers, or hyphens followed by x number of consonants in a row. We need the optional match because once you put the literal period in their the subdomain will have to start with the matched consonants, or it will fail.

If anyone is interested in testing this in the real world I created a subdomain of this website to help so you don’t have to wait for some user to get drive by redirected to a bad domain.  Just visit this link http://test.abbbbbbbbbba.scottfromsecurity.com (10 B’s) and then search your pcaps. Enjoy and feel free to leave a comment should you come up with something better or need to correct anything I’ve written above.

Final screenshot

final_regex_matches

Quick Tip: To get the extra data around query time shown in screenshot above, put ” /debug” after the path in the shortcut as shown.

NW Investigator shortcut with Debug switch

Debug switch on shortcut

PS.  In a related note, I saw this Snort signature looking for a DNS request for a hostname with 5 to 32 consonants in a row with a .cn TLD.  It was created around April this year at the same time as I was working on mine, I figured I’ll consider it great minds thinking alike.

—————————————————————————–

Event : INDICATOR-COMPROMISE Suspicious .cn dns query (1:15167), Timestamp : 2012-09-22 01:28:03 Classification : A Network Trojan was Detected Destination Port/ICMP Code : 53 (domain)/udp

Rule : alert udp $HOME_NET any -> $HOME_NET 53 (msg:”INDICATOR-COMPROMISE Suspicious .cn dns query”; flow:to_server; content:”|01 00 00 01 00 00 00 00 00 00|”; depth:10; offset:2; content:”|02|cn|00|”; distance:0; pcre:”/[x05-x20][bcdfghjklmnpqrstvwxyz]{5,32}[^x00]*?x02cnx00/i”; metadata:policy security-ips drop, service dns; classtype:trojan-activity; sid:15167; rev:11; )

Advertisements

2 thoughts on “RSA Netwitness Investigator Regular Expressions

  1. Hi,

    I was working on Netwitness and stuck in a query where I dont want to have Internal IP in the report. Ans Seriously I am not fining any solution please hep me

    Like

    • Hi Priyank,

      Well there might be a bit of a language barrier and I don’t have anything beyond the free local install of Netwitness anymore, but I’d need more info to try and help you with your question. Go ahead and use my contact me page on scottfromsecurity.com/cms/contact and I’ll see if I can help.

      Like

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s