What do you do all day?

That’s often the question I get asked by friends and family, and to be honest it’s hard to explain to non-techies so I figured I’d pick out a recent example of a typical malware find and cleanup at work and blog about it.
So it all started in the morning when I opened the daily report I email to myself from the proxy server that flags URL requests in categories I feel are malicious and accessed during the previous 24hrs.  Once blowing past the usual false positives one stood out to me, music.gr (TLD for Greece).  The report does not give full HTTP headers just the hostname so I had to look up the same hit in the proxy’s reporting server to get the time/date/username of the HTTP request.  Then I moved to the SIEM (log manager) to get correlated logs from FWs, netflow, and the parsed raw proxy logs showing all the fields with a start and end time +/- 5 mins.
In doing so I saw obvious bad outbound traffic from the suspect host with what looked like URLs centered around webmail.  This is not surprising as webmail is a significant weak spot in many  corporations, anyway here are some of the logs I was presented with, a bit sanitized.

Continue reading


Detecting the Bad from the Good…

UPDATE 4/3/12: I worked with Joel Elser on Snort-sigs mailing list to develop the below signature. However there’s been some concern around system resources of Regexing every GET request to the internet.  I’m thinking I might have to adjust the rule to exempt .com and .net TLDs.  Less effective I know but at least it won’t kill the sensor. This technique is probably better for offline static analysis of logs then realtime IDS.  Damballa has two good papers on their work around detecting DGA (Domain generation algos) and how they haven’t gone away now that Conficker is out of the news.  Links
alert tcp $HOME_NET any -> $EXTERNAL_NET $HTTP_PORTS (msg:”WEB-MISC http header with 9 or more consonants”; flow:to_server,established; content:”GET”; http_method; content:”Host: “; http_header; pcre:”/^Host:s[tnrshdlfcmgpwbvkxjyqz0-9]{9,}$/Hi”; metadata:service http; classtype:bad-unknown;)
UPDATE 10/28/12: The above sig does not match unless there are NO vowels in the host header.  Please see an updated post here
In security monitoring it’s your job to use your creativity to design rules and dashboards so you can identify evidence of malicious activity.  In general there are two strategic ways to do this, detect the bad (blacklisting) or detect the good (whitelisting).  It doesn’t take a genius to realize that whitelisting is the more effective strategy overall, although its much harder to implement.  Even so blacklisting is still useful to use for increasing defense in depth.   Speaking about getting creative to detect the bad guys, a recent thought I had is looking for one of the tactics they use to avoid being taken offline.  Namely registering and using a large number of domains so that content filtering and whitehat organizations can’t keep up.  You could see this during the Conficker worm battles, where early versions were programmed to connect to 250 domains a day, and when the Conficker Cabal launched an effort to pre-register those names later worm versions came out in direct response with algorithms for 50,000 domains per day.
The flaw I see in the “I can register more domains, faster then you can” tactic is often the make up of those domains. By their nature they frequently are a collection of puesdo random letters not valid strung together words in your language, like a normal domain.
For Example: (consonants in a row in red)

Countering the new threats…

The SANS Incident handling steps are

  • Preparation
  • Identification
  • Containment
  • Eradication
  • Recovery
  • Lessons Learned

I’m going to talk about Preparation and Identification today. See I enjoy the challenge of having a network and resources to protect.   Its funny to see how that translates into my non-work life, for example I always like RTS type games and often take a defensive posture by default.  Called turtling in the gaming world it’s more than likely to get you a loss because most games I played seem to favor the noob friendly kamikaze aggressive style, or rushing. However, that just makes me enjoy defense even more, who wants to take the uneducated inexperienced easy way out?  In the real world all out no fear, attack is not a legitimate info sec strategy for obvious reasons.

First it’s well documented how the “bad guys” have changed over the last 2 decades from notoriety seeking weekend hackers, to “hey I can make money at this” full time hackers, to organized criminal gangs.  What I don’t hear enough about is the current migration from gangs to an underground criminal marketplace, and that is just plain frightening.  Organized crime is dangerous and hard to stamp out but it’s a threat that can be met with equal good guy organization and cooperation.  Like when it took Elliot Ness and the federal government to stamp out the mobs and corruption of the 30’s.   Complicated Bad can be fought with complicated Good, hard but doable.  But how do you fight a decentralized economy of goods/service providers with a specialized skillset, profiting off loose, dynamic, and temporary connections to others?  That is really hard to do. Now throw in more recent fully state sponsored agencies targeting small subsets of the internet (Advanced Persistent Threat [APT] attacks on Google) and you’ll see why Information Security is so much in the news these days.

Continue reading

Blackhat SEO, The Next Generation…

UPDATE March 2012: So I was rereading my own blog and wondered if I got this post right.  I googled and turns out it was probably a FAIL.  Although Van Morrison deny’s it he bought Gigi a house in Texas and visited her and the child regularly (although she recently passed away).  Guess I should have known writing about celebrity gossip, the truth will probably never really made public.

December 29th 2009  I noticed a story come across the wires about the singer Van Morrison and Gigi Lee having a baby. This was picked up by the Associated Press and many legitimate news outlets.  Turns out it was a carefully orchestrated plan to drive traffic for keywords already seeded on hacked websites that redirected to mostly known fake AV malware servers (more on that at bottom).  Not knowing this at the time I did a quick google search out of normal user interest and got these results…

Being the paranoid security guy I am, I immediately noticed the similarity in the URLs  and that they weren’t domain’s of news sites.  For example domain.com/xxx.php?=gigi%20… or domain.com/xxx.php?=van%20…  Hmmmm, those don’t look like legit results to me.  Welcome to the world of Blackhat SEO, I don’t presume to be the end all authority on this as Dancho Danchev and others Sophos have been tracking this for years. But this was a new twist, the bad guys were not grabbing the currently hot top search results (like when a celebrity dies) and competing with other pages to get their rank high, they INVENTED the keywords  and already had the seeded keywords in Google’s page rank before attacking Van Morisson’s website!  Gotta to respect the ingenuity, wish they were on the good guys side. Whatever Google is doing to counter the bad guys from gaming their page rank algorithm it isn’t working very well, although in this instance Google was working as intended.  If a malware author can poison a person’s view of the web (search engine results) then the average user doesn’t have much of a chance.  Turns out any one of the links redirected me to a known malware page. I followed them with Malzilla, here’s an example…


1. First the click to Google’s search results
http [break] ://www.google.com/url?sa=t&source=web&ct=res&cd=17&ved=0CB8QFjAGOAo&url=


2. Redirects to the search result

HTTP/1.1 302 Found
Location: http [break]://xxxxxx-law.com/mvf.php?t=gigi%20lee
Cache-Control: private
Content-Type: text/html; charset=UTF-8
Date: Wed, 30 Dec 2009 17:04:04 GMT

Which is a .php script that looks like it’s taking an input of “t=gigi lee”  (the %20 is an encoded space) So I tried it w/o correct input and with “wget” default user-agent and was cleverly 301 redirected to cnn.com homepage, I thought that was a nice touch by the bad guys.

3. Continuing with the correct link gets me too, (hmmm  random .pl domain not a good sign.  No offence intended to Poland)

HTTP headers:

HTTP/1.1 302
Date: Wed, 30 Dec 2009 17:04:12 GMT
Content-Type: text/html
Server: Apache

Location: http [break]://vby1x4.xoeg .pl/in.php?t=cc&d=29-12-2009_tr2&h=xxxxxx-law.com&p=http%3A%2F%2Fwww.google.com%2Furl%3Fsa%3D…. snip….6vz49MerFAydtg

4. Then finally the below link which my endpoint HIPS stopped.  Just from the link you can tell it’s a fake AV Trojan and probably a couple exploits to go along with it (I didn’t go down the rabbit hole any farther).  Also I was impressed with Firefox WOT add-on (link) as I kept having to disable it to follow the redirects with FireFox.  Definitely recommend it, along with no_script of course

HTTP/1.1 302 Found
Date: Wed, 30 Dec 2009 19:08:52 GMT
Server: Apache/2.0.55 (Unix) PHP/5.2.1
X-Powered-By: PHP/5.2.1
Location: http [break] ://createpc-pcscan-kokn .net/?uid=195&pid=3&ttl=e11476d0489

So I was intrigued by this .php file that they were able to upload to many websites like the example above which is a law firm in Boston Mass. I contacted several of them to tell of the infection and to ask if I could get a copy of that server side .php script, but none have done so.

So the whole thing was an elaborate scheme to get hits as it was most likely the same group of hackers that compromised the singer’s website and started the whole thing.  I was wondering how they knew to upload the files with the right keywords before the news broke and had figured they must have ongoing access to adjust the keywords or replace the .php depending on the current news story, which still could be true. The AP picking up the story must have had the bad guys celebrating for sure.

From BBC news site:

The Belfast-born 64-year-old said he had been the victim of an internet hacking attack that had placed “falsehoods” on his official website. BBC News was one of several outlets to report the hoax as fact.”The comments which appeared on my website did not come from me,” he [Van Morrison] said, in a statement issued to the media. The singer said he had asked his management team to carry out an immediate investigation, adding it was the second time his website had been hacked in the last three months.

Link to MTV talking about it.  They missed the point though it wasn’t an innocent hoax, it was motovated by the second oldest story book…..Money.

Is it Malware?

One lesson I remember learning during my teen years was the world is really shades of gray, not so much the black and white it seems when you’re younger.  The same principal applies to malware, what really is…bad?  If an AV scanner asks for payment before clears your spyware cookies and removes other fake AV installs is it malware?  What if separate 3rdparty affiliates install it w/o your or manufacturer’s permissions, through an exploit, social engineering, or by forcing you to opt-out? What if they steal other companies detection DBs as in the iobit and malwarebytes saga? Apple installs file sharing software (Bonjour) w/o notification and with opt-out techniques (Google Toolbar, Quicktime) when installing iTunes. Does that make Apple a spyware purveyor?  Again, shades of gray.

So what is the user/IT tech to do? Well there’s no easy answer, and in my humble opinion you have to trust but verify with research.  Check already trusted forums/websites of whitehats and coworkers along with other trusted IT user’s opinions.  In short, do your homework before installing any software on your machine.

Along these lines an interesting thing happened while I was dealing with a small outbreak of Vundo.Trojan at work. Our AV vendor didn’t detect the sample yet so I recommended for the IT staff to install update and run malwarebytes in safemode. For various reasons one infected computer had no immediately available IT rep so it was left up to the user.  When getting to the download link he was tricked by a deceptive (my opinion) advertisement on download.com (Right hand “Bad Link” in screenshot below) to install a “shades of gray” program called CyberDefender instead of malwarebytes (a trusted whitehat community supported malware scanner).

Tricky Tricky for the average user

This program has a history of being considered actual malware, but apparently was taken over by different management whose is supposedly trying to legitimatize it circa 2006.  I decided to check it out a bit and am suspicious enough to recommend NOT installing it.

Some issues that make me dubious to install the software…

  • Inflates the severity of the findings (ie. Detected sysinternals processmonitor as a HIGH risk)
  • Opens a high tcp port and listens on it. (In 2 separate installs and 1 reboot on a clean VMware image I saw 1stcdas.exe listening on tcp/5710, 2ndcdas45.exe on tcp/5754, and 3rdon reboot saw tcp/5779 (from THE SAME installer file))
  • Advertises with shady deceptive ads
  • Finds different threats with a program uninstall/reinstall and subsequent rescan with Cyberdefender
  • Offered me to buy the product for 250.00 after an add/remove programs uninstall (clearly a ripoff)

Cheap Website Monitoring…

A big part of being a security analyst is figuring out why something you admin is blocking what a customer is trying to get to.  I actually like those problems because I look at them like a mini-puzzle. The key to these issues, and many others for that matter, is being able to recreate the trouble with your own equipment.  I usually tell people, if it’s blocked for me too then I will be able to fix it.  The hard problems come when it’s an occasional issue, or only from one part of the network, etc.

Anyway, the latest scenario involved a city employee trying to get to a local high school’s website to get their layout.  The site was blocked with the category pornography, which seemed like a miscategorization.  After recreating the problem on my desktop I got a hunch, which leads me to the reason for this post.  I headed to google and searched “site:xxxxxx.edu nude”  and it came back with the results that would make any webmaster wince.  Pictured below (anonimized to protect the innocent)…

So that was quickly solved by making the school’s webmaster aware of the injected HTML SEO poisoning keywords and asking our vendor to re-evaluate the site once cleaned.  But more to the point such Google searches are a really cheap way to do some manual monitoring for websites under your protection. I personally do searches like these every few weeks, on the off chance one day I will get something other then no webpages found.  Don’t forget to submit requests to clear the major search engine’s cache if you’re hit or these results will stick around for a while.

PS I’ll leave it up to the reader’s imagination on which keywords to use.

AutoRun part 3 and final…

So I heard through SANS ISC that M$ has decided to announce through their blogs that they are going to turn off autorun by default.  They didn’t say how exactly yet but I think it’s a good thing for all the regular users who don’t realize how dangerous that innocent thumb drive really is.  Link here…. I, nor the City of San Diego, has to worry about it as I got to have part 1 and part 2’s “learning experiences” which although frustrating were educational, and effectively turned off autorun for all PCs.

AutoRun/AutoPlay part 2…

This is an update to the “AutoRun/AutoPlay Disabling Confusion” blog entry below. I’ve been working with the WSUS admin and we have not gotten the KB950582 to install on machines below Vista. Not good.  I even checked several C$ machine shares and did not see the patch’s install folder, proving the “not applicable” option in screen
shot below.  Again Not good.

WSUS showing patch as either applied or not applicable..

However, just yesterday I logged onto one of my XP machines and I got a Windows update rompt, but that was odd because I was sure I was up-to-date. The prompt says you need  B967715 which is a patch that correctly disables AutoRun. Nice! So, MS did simplify the whole thing. They released Security advisory 967940 which points to patch 967715 for 2000,XP,2003. And to top it all off MS even addresses the confusion I mentioned in their FAQs. So my new suggestion is to apply the GPO and include the 967715 to your WSUS required patches and you should be good to go. Thanks for addressing the issue MS, maybe Bill Gates reads my blog?

Here are the links…



PS. If you already installed KB950582, you won’t need the KB967715 and it won’t show as needed.

Why are there two places to get this update?

These updates are available in two places due to the way the updates were originally offered. The updates that were offered in Microsoft Knowledge Base Article 953252 were not available from automatic updating (including Automatic Updates, Windows Update,
and Windows Server Update Services) and therefore required users to manually find these updates and install them. The updates that are offered in Microsoft Knowledge Base Article 967715 contain the same updates that correctly respect the registry keys values to disable Autorun as in Microsoft Knowledge Base Article 953252, but are being distributed via automatic updating.

AutoRun/AutoPlay Disabling Confusion

So I have been working on disabling AutoRun/AutoPlay, which I will just call AutoRun from now on, although they are technically not the same thing, as far as disabling them they are.  Microsoft, in my humble opinion, makes this confusing for no necessary reason.  Especially since malware executing from external drives (other then installed HDDs) is easily the 2nd highest vector for the bad stuff getting onto San Diego City computers. So the need to disable AutoRun is a no brainer, after all it’s been stopped for A: drives and can’t even be allowed if desired. What makes a USB, CD, DVD, Network Drive, etc any different? The currently spreading Downadup/Conficker worm would be very happy for system admins/users to allow the admittedly convenient AutoRun functionality despite how dangerous it is.

AutoRun has 3 parts, and they must all be disabled or you won’t be safe from it’s unchecked executing. The 3 parts are….

*  User Double Clicks (in the MyComputer view, NOT explorer or folders view)
*  Contextual Menu (Right Click menu)
*  AutoRun (Automatically run anything that is listed in autorun.inf on “new” drive)

The setting below (done through registry or GPO [steps here])

NoDriveTypeAutoRun (set to FF or all drives)

should be all you need to do; to be AutoRun free, right? Wrong.  Turns out this will only disable the AutoRun parsing of new drives. But it will NOT disable the first 2 in my list above, which means the average user will just probably launch the malware manually. So what do you do?  Well you turn to MS to explain why they aren’t respecting the GPO you just pushed. Unfortunately you will just be met with more confusion. Here’s the link you will find for anything older then Vista/Win2008.


So you scroll to the bottom to download the patch, but notice that it’s really a link to KB950582, that is labeled as Vista/Win2008. What the heck? Then you go to your WSUS console to download the correct patch for WinXP/2000/2003 etc. but it’s not there. Only KB950582 is, which by the way was a security patch (MS08-038) for Vista/Win2008 released back in July 08. Long story short what I’ve been able to put together is that the patch KB950582 when applied to other Windows OS’s modifies the shell in a way that allows for honoring the AutoRun registry entry for earlier OS ‘s AND repairs a vuln in Vista/Win2008. Which is why its labeled as Vista/2008 and not XP/2000/2003, but I think MS is wrong to not make it clear and simple to understand what needs to be done to disable AutoRun/AutoPlay for all the Windows OS flavors. They are usually pretty good with scanning and giving your computer the patches it needs, lets hope they decide to change their approach in the near future.

My Solution: Set the GPO for HK Local Machine in Active Directory, and then include the patch KB950582 in WSUS for all your machines.  It should push out even though it says Vista/2008.  I’ll update this blog if I find this to not be true.

UPDATE 2: So I’m still on this and it appears WSUS is NOT pushing KB950582 to non-Vista machines (see next blog post)

PS. US CERT recently released a bulletin recommending to disable AutoRun, and detailed a fix from Nick Brown (which is great for home but probably a little severe for the Enterprise, IMO). They also put in an update about what I explained above you might want to check out there short “update” paragraph at the bottom if my opinionated ramblings above didn’t make 100% sense.


Deleting Files Already in Use Remotely…

I had a bit of a frustrating day the other week, a machine was infected with an IRC bot that AV couldn’t detect or remove, beyond the relatively minor host file modifications. The machine was re-imaged and then even replaced, but kept getting reinfected a few hours later.  Of course 99% of the time that means a PEBCAK (google it).  It turned out to be a user’s personal infected thumb drive that was spreading the bad stuff.  Anyway since we were getting a large amount of email alerts from the IPS on outgoing IRC NICK registration to IPs in Asia, I RPDed to the machine and copied TCPview from a share to see what exe was creating the process sending the IRC traffic.  The malware was able to hide the process from taskmgr but not the sysinternals tool. It turned out to be the file run64dll.exe so I tried to stop the process and it immediately started back up, not surprising.  I tried the delete the exe but Windows complained about it being in use.  This is usually where safe mode comes into play but I was sitting miles away, so I employed a trick that is the reason I’m writing this blog entry.  Of course you will most likely have to be local Admin or have those rights on the target folder/file, but when you need to kill a file in use try changing the permissions on the file to DENY ALL (Right click file and go to permissions tab in WinExplorer).  This will then quickly stop the process you couldn’t from accessing the file and probably will allow you to delete the file.  Worked for me and hopefully it will for you.  After that I highly recommend a full re-image, you have proof that malicious software successfully ran on that machine and there is no way to know for sure how deep the infection went, don’t risk it just wipe it.