01.14.08
Port Scanner Challenge: And the Winner is?
The other day I posted raw data comparing nmap, PortBunny and Unicornscan... I thought today I'd provide some of my thoughts on what the data shows us.
In the end I scanned 5 hosts running a variety of operating systems and I think I gave a fairly decent small scale spread and one initial comment I'd like to make is on the scanning of the HP LaserJet 4MV... While not all scanners found all the ports, they were all able to scan it... which I found fairly impressive... especially considering I've crashed it numerous times in the past playing with advanced options in port scanners and packet creation programs.
Now for the anlysis... Was there a winner? At first I didn't think so... but once I created the graph it became fairly evident that there was. Before I declare the winner... let's take a look at what we saw.
Unicornscan
I was fairly impressed with unicornscan the first time it ran... at least from a speed standpoint. That is until I ran nmap and PortBunny. While unicornscan (on a standard scan, default ports) was able to provide consistent speeds... it was clear that on systems with fewer open ports... there was a huge disadvantage in the design of unicornscan... The consistent speeds were still occurring. If we look at my shell box for example (Ubuntu 6.06 PPC on an old 350Mhz iMac), we see that unicornscan took what appears to be a respectable 9.2 seconds. However, nmap took only 1.5 seconds and PortBunny was less than a second at 0.7 seconds.
The full port scan also didn't bode well for unicornscan. On two hosts, the printer and the gateway, it failed to find any open ports... These are both slower systems (the older printer, and a soekris 486 for the gateway) so perhaps they couldn't keep up with the speed of the scan... or perhaps unicornscan was scanning too fast even for itself.
In the end, after seeing the results of nmap and PortBunny I was rather unimpressed with unicornscan.
nmap
I was quite impressed with nmap. During the default scan it tied PortBunny with lowest number of missed ports (of course this is due primarily to the various scanners default port list) and on a full port scan, it was the only scanner to find all open ports. In addition to having the lowest miss port rate... it boasted the fastest times... coming in 17.5 minutes faster than PortBunny on a full port scan, and 16 seconds faster on a default scan.
Additionally when I provided nmap with the '-T5 --max-retries 0' options, it blew PortBunny out of the water... it missed two additional ports over PortBunny on one of the 5 hosts, however the time difference was 5.3 seconds to 74 seconds... nmap was 68.7 seconds faster.
PortBunny
Given all the hype surrounding PortBunny and the fast that is is a "Linux Kernel-Based Port Scanner" (which is supposed to work to it's benefit), I was expecting great things... instead I was seriously disappointed. There wasn't a single scan where PortBunny fully out performed nmap... You have to set nmap to get ridiculous scan speed (scanning "almost too fast") in order for PortBunny to even manage to find more ports than nmap and then it takes ~15 times longer to find those 2 extra ports... Without those "almost too fast" options, nmap still performs faster than PortBunny and with more accuracy.
There was one host where PortBunny was able to outperform nmap, however that was with nmap doing a default scan... when timing options were adjusted, once again PortBunny failed to beat nmap.
Decision
When I started this challenge, I wasn't sure what the outcome would be... the only prediction I had was that unicornscan would be defeated by both PortBunny and nmap. This proved to be true... Between nmap and PortBunny, due to the hype around PortBunny and the claims that I had seen... I really wasn't sure. I expected it to be a close battle between the two... at most a TKO... but in the end it was a straight-up KO and in reality PortBunny was never really a contender.
Robert E. Lee said,
January 15, 2008 at 3:51 am
Unicornscan tends to have better results when you have more experience using it
Unicornscan’s biggest weakness is that it does not currently do logistics/timing for you. It is up to the tester to figure out how much bandwidth or how many pps is safe to send at.
By default, scanning all 65k ports should take ~3 minutes, 45 seconds at 300 pps. 300 pps is the default speed that unicornscan goes unless you request a faster speed with the -r flag.
If it was only taking 9 seconds, and you were not requesting a higher speed, then I’d guess that the TSC on your computer isn’t functioning properly. If the TSC isn’t working properly, all of your scans will suffer from dropped packets. Try again with -d2 (gtod timer).
Nmap, portbunny, and unicornscan have very different methods of accomplishing what they’re doing. I could run “heads up tests” that show each to be the clear winner, depending on what metrics I choose to measure. This particular comparison wasn’t an accurate representation of any of the projects.
–Robert
Tyler Reguly said,
January 15, 2008 at 4:17 am
Thanks for the comments Robert.
I’ll admit that I’m not overly experienced with Unicornscan… That being said… One aspect of a scanner that has to be considered is usability, since most network / security people don’t have time to figure out the settings available for the scanners and learn how to best use them in different situations… which was why I did a default scan for each, which was directly out of the unicornscan documentation. I added the default options for nmap only because it is, as far as I know, the only scanner that performs retries by default and that is somewhat throttled. (Other than unicornscan being limited to 300pps).
Personally, I wanted to see what would happen if I ran the scans in default modes… in the end… I think any scanner will be hard pressed to beat nmap… but that being said… I invite the authors of each of the three scanners to provide me with what they consider to be the “ultimate” scan settings for their scanners. I’ll perform additional scans with each provided setting and provide an updated chart with the data from the “tuned” scans.
As an example… I can shave 11 seconds of of nmap’s 14 second default scan time against the vista box, simply by providing the -sS flag, which I probably should have done but I wanted to run everything “out of the box”
Robert E. Lee said,
January 31, 2008 at 11:36 am
Here is an another review of the 3 scanners:
http://loquens-caesu.blogspot.com/2008/01/port-scanner-challenge-revisited-nmap.html
Harry said,
February 4, 2008 at 4:05 am
I would take accuracy over speed any day of the week and personally would not rate a port scanner by how fast it can scan ports - unless it takes an absurdly long time of course. (I’m presuming you’ve never had to try and avoid an IDS/IPS?)
If a network admin or a ’security person’ as you put it can’t find time to learn how to use a port scanner then they are in the wrong trade - likewise any professional who is using a port scanner in such a way that he may need to use a certain scanner for a certain task will not be using default settings for default scans……a kid in school may do so, but not someone who needs it for their job.
UnicornScan in certain situations massively exceeds Nmap - and vice versa - I can’t speak for Port Bunny as I have never used it.- it all depends on your goal and how you need to achieve it.
.:Computer Defense:. » Update on Port Scanner Challenge said,
February 4, 2008 at 4:27 am
[...] other day I posted the Port Scanner Challenge, and a follow-up article declaring a [...]
Tyler Reguly said,
February 4, 2008 at 4:33 am
@Harry,
I would also take accuracy as well… but within limits… There are times when I just want the scan done, and in those places… I’ll take the speed with reasonably accurate results. Also avoiding an IDS/IPS has nothing to do with straight scanning a host which is what I was looking at.
I think you would be quite surprised at how many people, both network admins and ’security people’, don’t learn all the finer points of the tool. It has nothing to do with being in the wrong trade. It has to do with requirements and how many intricacies some of these tools have. I was looking at straight point and shoot and I’ve seen plenty of people in plenty of situations (most situations in fact) do just that. I would argue that the kid is school is more likely the explore the options than a network admin. I don’t know where you work, but in a lot of places you don’t have time to sit and fiddle and learn how to tweak a tool… You especially can’t justify it when there’s another tool that will do just as good of a job straight out of the box… which is one of the problems I have with UnicornScan.
Harry said,
February 5, 2008 at 4:39 am
Where I work if you didn’t look into tweaking a port scan and relied on a default scan completing a fast as possible, then you wouldn’t be in the company for very long.
I am presuming you don’t have much Pen testing experience. I can’t remember the last time I have ran a default low port scan and trusted the output enought to declare the box secure in my report, without using other tools with a variety of diferent options, espeically when scanning a host over the internet.
I think your test is very unrealistic and is typical of a ‘Skiript Kiddie’ port scan or a scan of someone who does not need 100% accurate and complete information - try runing mnap against multiple hosts on multiple networks behind any top brand IDS - and then try running unicornscan against the same hosts sitting behind the same IDS.
You would come out with a clear winner and it would not neccesarily be Nmap.
Christoffer Strömblad said,
February 8, 2008 at 10:57 am
Personally I think these tests have little relevance and are confusing at best. They really don’t show anything, nor do they help improve anything. It all comes down to personal taste. More often than not, the people using the tools don’t have sufficient knowledge to actually perform any sort of testing to demonstrate whether this tool is better than the other.
The most qualified people would probably be the developer themselves. Let them all agree to some metrics, and then conduct testing with each tool in their own environments.
Even with this testing it will prove little… because the idea of comparing them in the first place is pointless. Anyways… now I’m starting to rant, so I’ll just leave it at this.
Stop arguing and waste time… do something to improve the situation instead.