When it comes right down to it, in both the Apache and OpenSSH cases, the vulnerabilities were patched in a very short period of time. But I have a number of questions about what's been going on. But first, let's look at the Apache problem. This link leads to the initial advisory, from the ISS site. It was issued widely across the net, including a posting to BugTraq. Here's the synopsis:
ISS X-Force has discovered a serious vulnerability in the default version of Apache HTTP Server. Apache is the most popular Web server and is used on over half of all Web servers on the Internet. It may be possible for remote attackers to exploit this vulnerability to compromise Apache Web servers. Successful exploitation may lead to modified Web content, denial of service, or further compromise.
One small problem, or maybe more... First, the Apache Foundation developers had already found the problem while exploring another issue, and were working on it. ISS might have learned this had they bothered to have a dialog with the Apache people prior to their "Full Disclosure". One other little problem with their advisory is that they [ISS] also developed and released a source patch for Apache, independently of and without the advice or testing of the Apache developers. That patch did not adequately address the problem, nor was it appropriate for them to do that.
The Apache Foundation then quickly addressed the issue themselves in this release, including this introduction:
While testing for Oracle vulnerabilities, Mark Litchfield discovered a denial of service attack for Apache on Windows. Investigation by the Apache Software Foundation showed that this issue has a wider scope, which on some platforms results in a denial of service vulnerability, while on some other platforms presents a potential a remote exploit vulnerability.
We were also notified today by ISS that they had published the same issue which has forced the early release of this advisory.
The Common Vulnerabilities and Exposures project (cve.mitre.org) has assigned the name CAN-2002-0392 to this issue.
And the fur began to fly. ISS backpedalled a bit, but remained aggressive, while the Apache people continued to hack away at a real solution. Within 24 hourse, patched versions in both the 1.X and 2.X trees were available for vendors to use in release security-updated packages to their distribution customer base. Gentoo and Debian were out the same day. I'll assume that other distros weren't far behind. Here's the latest announcement on the Apache site:
UPDATE: (supersedes security bulletin 20020617)
This follow-up to our earlier advisory is to warn of known-exploitable conditions related to this vulnerability on both 64-bit platforms and 32-bit platforms alike. Though we previously reported that 32-bit platforms were not remotely exploitable, it has since been proven by Gobbles that certain conditions allowing exploitation do exist.
Successful exploitation of this vulnerability can lead to the execution of arbitrary code on the server with the permissions of the web server child process. This can facilitate the further exploitation of vulnerabilities unrelated to Apache on the local system, potentially allowing the intruder root access.
Note that early patches for this issue released by ISS and others do not address its full scope.
Due to the existence of exploits circulating in the wild for some platforms, the risk is considered high. The Apache Software Foundation has released versions 1.3.26 and 2.0.39 that address and fix this issue, and all users are urged to upgrade immediately. These versions are available for download; see below.
OK. So that was last week. What's going on now?
On this last Monday (the 24th), Theo de Raadt, lead developer for OpenBSD and OpenSSH, announced an unspecified OpenSSH vulnerability. Here's his letter in its entirety:
There is an upcoming OpenSSH vulnerability that we're working on with ISS. Details will be published early next week.
However, I can say that when OpenSSH's sshd(8) is running with priv seperation, the bug cannot be exploited.
OpenSSH 3.3p was released a few days ago, with various improvements but in particular, it significantly improves the Linux and Solaris support for priv sep. However, it is not yet perfect. Compression is disabled on some systems, and the many varieties of PAM are causing major headaches.
However, everyone should update to OpenSSH 3.3 immediately, and enable priv seperation in their ssh daemons, by setting this in your /etc/ssh/sshd_config file:
UsePrivilegeSeparation yesDepending on what your system is, privsep may break some ssh functionality. However, with privsep turned on, you are immune from at least one remote hole. Understand?
3.3 does not contain a fix for this upcoming bug.
If priv seperation does not work on your operating system, you need to work with your vendor so that we get patches to make it work on your system. Our developers are swamped enough without trying to support the myriad of PAM and other issues which exist in various systems. You must call on your vendors to help us.
Basically, OpenSSH sshd(8) is something like 27000 lines of code. A lot of that runs as root. But when UsePrivilegeSeparation is enabled, the daemon splits into two parts. A part containing about 2500 lines of code remains as root, and the rest of the code is shoved into a chroot-jail without any privs. This makes the daemon less vulnerable to attack.
We've been trying to warn vendors about 3.3 and the need for privsep, but they really have not heeded our call for assistance. They have basically ignored us. Some, like Alan Cox, even went further stating that privsep was not being worked on because "Nobody provided any info which proves the problem, and many people dont trust you theo" and suggested I "might be feeding everyone a trojan" (I think I'll publish that letter -- it is just so funny). HP's representative was downright rude, but that is OK because Compaq is retiring him. Except for Solar Designer, I think none of them has helped the OpenSSH portable developers make privsep work better on their systems. Apparently Solar Designer is the only person who understands the need for this stuff.
So, if vendors would JUMP and get it working better, and send us patches IMMEDIATELY, we can perhaps make a 3.3.1p release on Friday which supports these systems better. So send patches by Thursday night please. Then on Tuesday or Wednesday the complete bug report with patches (and exploits soon after I am sure) will hit BUGTRAQ.
Let me repeat: even if the bug exists in a privsep'd sshd, it is not exploitable. Clearly we cannot yet publish what the bug is, or provide anyone with the real patch, but we can try to get maximum deployement of privsep, and therefore make it hurt less when the problem is published.
So please push your vendor to get us maximally working privsep patches as soon as possible!
We've given most vendors since Friday last week until Thursday to get privsep working well for you so that when the announcement comes out next week their customers are immunized. That is nearly a full week (but they have already wasted a weekend and a Monday). Really I think this is the best we can hope to do (this thing will eventually leak, at which point the details will be published).
Customers can judge their vendors by how they respond to this issue.
OpenBSD and NetBSD users should also update to OpenSSH 3.3 right away. On OpenBSD privsep works flawlessly, and I have reports that is also true on NetBSD. All other systems appear to have minor or major weaknesses when this code is running.
This announcement raised a number of of questions for which there was no good answer. We can't evaluate the danger or type of the potential hole, and it seems on the face of it that Theo is using this problem to "market" his latest addition to the OpenSSH codebase: Privilege Separation. While it's a good thing, it hasn't been around for a long time, and there's little practical experience running priv sep. Is this cure going to be worse than this disease? And why is Theo being such an apparent butthead about the various distributions and some of their specific personnel? All of these and many other statements flew back and forth on several of the mailing lists I hang out on. But one thing caught my eye... "There is an upcoming OpenSSH vulnerability that we're working on with ISS."
Wait a minute, isn't that the group that blindsided Apache just last week?
According to their website, ISS is Internet Security Systems. I hadn't heard of them before this last few weeks. Certainly not one of the big boys, until all this recent press. From their marketing crap on the homepage (http://www.iss.net), it appears they are in the same biz as McAfee and Norton, but at a different tier. But wait, here's something interesting:
Bindview, Foundstone, Guardent, @Stake, and Internet Security Systems joined with [Microsoft] to declare they would immediately begin following a policy of limited public disclosure of security vulnerability information. Members of the coalition who discover new vulnerabilities will omit from their initial public advisories any details about how a hole might be exploited in an attack, and will not include code that demonstrates the bug. Thirty days after the first advisory, a more detailed noticed can be released under the rules.
This is found on the Security Focus website, an article entitled: Microsoft Reveals Anti-Disclosure Plan. Mmmm.
So ISS is a partner of Microsoft's? And they've helped out in two bits of Open Source news that broke in a very awkward way over the last couple of weeks. Are there any speculations that I can draw from that? Did someone get paid? Did they pressure Theo (who is apparently not known for his political and social acumen) into jumping the gun, holding the spectre of the recently embarrassed Apache group over his head? Why isn't ISS following the guidelines of the group they helped to form? Do they "have it in for Open Source"? They'll deny any such allegation, of course. I have nothing to back up my speculation, except their involvement and behaviour.
It's important to note that there are vulnerabilities in every piece of software devised. In order to work, the network has to be connected and communicating. The only remotely secure machine is one not connected to the net, unplugged and locked in a vault. But do you trust the guy who stands outside the vault with a gun on his hip?
I'll tell you this much. I trust my systems, and my administration of them, because I know as well as I can, what effect my actions have on the software I run. I would have no such assurance if I were running closed-source code. I trust the open source development process to work. I don't trust closed-source vendors to do anything but promote the next release that creates more revenue.
GNU Emacs version 21.1.1 was used in the construction of this article, running on a Gentoo Linux v1.3a workstation, running GCC3.1 and playing out on the bleeding edge once again. Presentation of the articles here is courtesy of LinuxMuse and the MagaMuse software written by Greg Lincoln. I just clean the toilets. I've been told that I can use a toothbrush next time, if I get them clean enough.
This article is Copyright 2002 by Brian P. Bilbrey. All rights reserved. Brian is a soon-to-be-Maryland-based geek looking for full time employment (read the resume here), an author, administrator, technical writer, product designer and husband. He enjoys reading, fishing and hiking, but is usually found behind a keyboard or three instead. It's not his fault, he watched too much British comedy on television during those all-important formative years.
Linux is a registered trademark of Linus Torvalds. Linux systems contain a large component of GNU Software, see www.gnu.org for details.
All other brand and product names are or may be trademarks of, and are used to identify the products and services of their respective owners.
All other content Copyright (C) 2002 Linux Muse. Powered by MagaMuse v0.3.5, (C) 2002 Greg Lincoln.