Program exploitation is a staple of hacking. As demonstrated in the previous chapter, a program is made up of a complex set of rules following a certain execution flow that ultimately tells the computer what to do. Exploiting a program is simply a clever way of getting the computer to do what you want it to do, even if the currently running program was designed to prevent that action. Since a program can really only do what it’s designed to do, the security holes are actually flaws or oversights in the design of the program or the environment the program is running in. It takes a creative mind to find these holes and to write programs that compensate for them. Sometimes these holes are the products of relatively obvious programmer errors, but there are some less obvious errors that have given birth to more complex exploit techniques that can be applied in many different places.
A program can only do what it’s programmed to do, to the letter of the law. Unfortunately, what’s written doesn’t always coincide with what the programmer intended the program to do. This principle can be explained with a joke:
A man is walking through the woods, and he finds a magic lamp on the ground. Instinctively, he picks the lamp up, rubs the side of it with his sleeve, and out pops a genie. The genie thanks the man for freeing him, and offers to grant him three wishes. The man is ecstatic and knows exactly what he wants.
“First,” says the man, “I want a billion dollars.”
The genie snaps his fingers and a briefcase full of money materializes out of thin air.
The man is wide eyed in amazement and continues, “Next, I want a Ferrari.”
The genie snaps his fingers and a Ferrari appears from a puff of smoke.
The man continues, “Finally, I want to be irresistible to women.”
The genie snaps his fingers and the man turns into a box of chocolates.
Just as the man’s final wish was granted based on what he said, rather than what he was thinking, a program will follow its instructions exactly, and the results aren’t always what the programmer intended. Sometimes the repercussions can be catastrophic.
Programmers are human, and sometimes what they write isn’t exactly what they mean. For example, one common programming error is called an off-by-one error. As the name implies, it’s an error where the programmer has miscounted by one. This happens more often than you might think, and it is best illustrated with a question: If you’re building a 100-foot fence, with fence posts spaced 10 feet apart, how many fence posts do you need? The obvious answer is 10 fence posts, but this is incorrect, since you actually need 11. This type of off-by-one error is commonly called a fencepost error, and it occurs when a programmer mistakenly counts items instead of spaces between items, or vice versa. Another example is when a programmer is trying to select a range of numbers or items for processing, such as items N through M. If N = 5 and M = 17, how many items are there to process? The obvious answer is M – N, or 17 – 5 = 12 items. But this is incorrect, since there are actually M – N + 1 items, for a total of 13 items. This may seem counterintuitive at first glance, because it is, and that’s exactly why these errors happen.
Often, fencepost errors go unnoticed because programs aren’t tested for every single possibility, and the effects of a fencepost error don’t generally occur during normal program execution. However, when the program is fed the input that makes the effects of the error manifest, the consequences of the error can have an avalanche effect on the rest of the program logic. When properly exploited, an off-by-one error can cause a seemingly secure program to become a security vulnerability.
One classic example of this is OpenSSH, which is meant to be a secure terminal communication program suite, designed to replace insecure and unencrypted services such as telnet, rsh, and rcp. However, there was an off-by-one error in the channel-allocation code that was heavily exploited. Specifically, the code included an if statement that read:
if (id <: 0 || id > channels_alloc) {
It should have been
if (id < 0 || id >= channels_alloc) {
In plain English, the code reads If the ID is less than 0 or the ID is greater than the channels allocated, do the following stuff, when it should have been If the ID is less than 0 or the ID is greater than or equal to the channels allocated, do the following stuff.
This simple off-by-one error allowed further exploitation of the program, so that a normal user authenticating and logging in could gain full administrative rights to the system. This type of functionality certainly wasn’t what the programmers had intended for a secure program like OpenSSH, but a computer can only do what it’s told.
Another situation that seems to breed exploitable programmer errors is when a program is quickly modified to expand its functionality. While this increase in functionality makes the program more marketable and increases its value, it also increases the program’s complexity, which increases the chances of an oversight. Microsoft’s IIS webserver program is designed to serve static and interactive web content to users. In order to accomplish this, the program must allow users to read, write, and execute programs and files within certain directories; however, this functionality must be limited to those particular directories. Without this limitation, users would have full control of the system, which is obviously undesirable from a security perspective. To prevent this situation, the program has path-checking code designed to prevent users from using the backslash character to traverse backward through the directory tree and enter other directories.
With the addition of support for the Unicode character set, though, the complexity of the program continued to increase. Unicode is a double-byte character set designed to provide characters for every language, including Chinese and Arabic. By using two bytes for each character instead of just one, Unicode allows for tens of thousands of possible characters, as opposed to the few hundred allowed by single-byte characters. This additional complexity means that there are now multiple representations of the backslash character. For example, %5c in Unicode translates to the backslash character, but this translation was done after the path-checking code had run. So by using %5c instead of \, it was indeed possible to traverse directories, allowing the aforementioned security dangers. Both the Sadmind worm and the CodeRed worm used this type of Unicode conversion oversight to deface web pages.
A related example of this letter-of-the-law principle used outside the realm of computer programming is the LaMacchia Loophole. Just like the rules of a computer program, the US legal system sometimes has rules that don’t say exactly what their creators intended, and like a computer program exploit, these legal loopholes can be used to sidestep the intent of the law. Near the end of 1993, a 21-year-old computer hacker and student at MIT named David LaMacchia set up a bulletin board system called Cynosure for the purposes of software piracy. Those who had software to give would upload it, and those who wanted software would download it. The service was only online for about six weeks, but it generated heavy network traffic worldwide, which eventually attracted the attention of university and federal authorities. Software companies claimed that they lost one million dollars as a result of Cynosure, and a federal grand jury charged LaMacchia with one count of conspiring with unknown persons to violate the wire fraud statue. However, the charge was dismissed because what LaMacchia was alleged to have done wasn’t criminal conduct under the Copyright Act, since the infringement was not for the purpose of commercial advantage or private financial gain. Apparently, the lawmakers had never anticipated that someone might engage in these types of activities with a motive other than personal financial gain. (Congress closed this loophole in 1997 with the No Electronic Theft Act.) Even though this example doesn’t involve the exploiting of a computer program, the judges and courts can be thought of as computers executing the program of the legal system as it was written. The abstract concepts of hacking transcend computing and can be applied to many other aspects of life that involve complex systems.