Wednesday, October 24, 2012

Application Security of Core Banking Systems - A first reality check

Nowadays the constant discovery of new vulnerabilities and an increasing interest from the public puts pressure on software vendors to improve their products. Also with a rapidly growing number of security incidents, companies and organisations start to realise that the aftermath of security breaches related to vulnerabilities in proprietary and 3rd party applications causes considerable costs. Therefore an increasing number of organisations in the public and private sector acknowledge the importance of application security and start to make it an evaluation criteria for software procurement.

An area that is rather uncharted in terms of its level of commitment to IT security is the domain of Core Banking Systems (CBS). Who has ever heard of products like Avaloq, Flexcube or T24? If you are not working in the finance industry the chances are not very high that you have. The terminology refers to software that enables a bank to merge information technology to suit with its core needs of banking. Credit and loan-processing, account and transaction management, interest calculations and performing payments are just a few examples for the powerful and wide ranging capabilities of these systems. CBS products incorporate many daily operations of a financial organisation therefore the protection and the security of these systems require the highest priority.

Considering the importance of CBS products not only for a single financial organisation but for the entire financial ecosystem, SEC Consult decided together with Capgemini to shed some light into the maturity of CBS products in terms of application security. The study was written, conducted and compiled over a period of nineteen months, consisting of two parts, a vendor survey and a security test for selected CBS products. Although most of the selected vendors (Avaloq, FIS (with two products), Infosys, Misys, Oracle, SunGard, TCS Financial Solutions) participated in the study, there were unfortunately some, which did not take part (Callata├┐ & Wouters, Delta Informatique, SAP and Temenos). This article focuses on the security testing part of the study, illustrating CBS vendor’s pretense about their security testing approach and their findings. Additionally results of our own application security tests of selected CBS products will be depicted.

Part 1: The survey


We asked CBS vendors about their promises, commitments and relevant activities relating to the application security of their product. A detailed questionnaire was sent to all vendors and we recommended that the person responsible for IT security should answer the questions or at least conduct a quality assurance exercise of the questions and answers. The questions can be categorised in the following areas.

      Level of commitment to application security
      Management of information security
      Training for developers in application security
      Methods of secure development
      Threat modelling and security requirements
      Size and complexity of CBS product
      Security Testing & Identified Vulnerabilities
      Security Incident Response
      Standards and best practices for application security

One of the first findings we made when analyzing the questionnaire is that all of the vendors have an unquestionable commitment to state-of-the-art application security. The vendors describe the maturity of their CBS products in terms of application security as high, mature or highly sophisticated. The industry recognizes application security as an important objective and that is great news for anybody who uses CBS products and ever wondered if they are secure.

In a typical software development lifecycle the security testing phase is one of the last possibilities to identify gaps in terms of security requirements.  In order to test if the desired assurance level for application security is achieved, three levels of test depth can be applied.



The answers from the vendors regarding their security test approach vary significantly from each other. On one side of the spectrum are vendors who claim to have full coverage of their code base using all three test approaches and on the other end there are the vendors who do not cover the majority of the code base or even do not perform security tests with medium or high assurance levels at all.



An obvious conclusion to these findings is that a low assurance level for CBS products is clearly not sufficient, therefore some of the vendors have to step up their security testing efforts. Vendors also have to be prepared to put their claims to the test. Banks are advised to actively test the security of CBS products and include detailed security testing as an important aspect of a product evaluation process.

The collected data about security vulnerabilities identified by vendors in their CBS products varies considerably in terms of the quantity of discovered security weaknesses. Some vendors indicated that they have not encountered a single vulnerability during their security tests; others said that they have found more than 100. During our many years of performing application security tests we virtually never had an application where we did not have at least any low impact findings at the initial test. Having no findings at all is a strong indicator for the low quality of the performed application security tests. For the sake of fairness it has to be said that having hundreds or even thousands of findings does not necessarily proof a high quality security test either. Especially automated tools are notorious for producing a lot of false positives. For this reason not only the volume of found vulnerabilities has to be taken into account but also the quality of the results.

Part 2: CBS security testing        


In order to validate the results of the survey we originally intended to conduct the second part of the study by performing application security tests for the vendor products we had surveyed. So we offered a free of charge application security test to be conducted by our security experts on a test system provided by the vendor. Certain vendors showed a serious interest in participating but unfortunately, after certain deliberations, all vendors declined to participate.

While developing an alternative approach to the second part of the study we managed to gain support from the financial service industry. Fortunately, three banks allowed us to have a look at their CBS systems, which were already implemented, by the banks in question. As a result at least three of the products described in part 1 have been included in the second part of the study.

All of the three CBS products were tested using a blackbox approach. For the security tests only low-privileged users were provided. For all of three CBS products severe and critical vulnerabilities had been discovered. None of the vulnerabilities had been discovered by the quality assurance process of the vendors. The following types of vulnerabilities and the resulting technical impact have been discovered during the application security tests:

  • Cross Site Scripting (XSS) - Stealing the identity and spy a CBS user 
  • Privilege Escalation - Become a more powerful CBS user
  • Weak encryption - Stealing the password of a CBS user 
  • SQL Injection - Direct access to the database 
  • Direct OS Command Execution - Remote control of the server of the CBS

All of the found vulnerabilities were fully exploitable and proof of concepts including videos were created for each of them. In order to protect the participating banks as well as other customers we will not disclose any details regarding the vulnerabilities, the tested products and the participating banks.

Conclusion


During a decade of working in the IT security industry we have seen it so many times in the past with our clients and also software vendors we are working with - security does not happen overnight. It is a process and for those software vendors who did not have security in mind at the time they designed the software or during the software development phase they usually start at the bottom and the bottom means no or very low security. CBS products are found usually in a bank's internal network, but as these systems incorporate more and more functionality and serve as backend systems for Internet facing systems the attack surface is definitely growing. The results of our study have shown that at least some CBS vendors have to significantly improve the application security of their products. Further owners of CBS products are strongly advised to actively validate their products and to incorporate security requirements in any future procurement process. 

How to get the study?

For the complete study, send an email to cbs2012@sec-consult.com or visit https://www.sec-consult.com/en/Vulnerability-Lab/Studies.htm

Thursday, October 18, 2012

Are web application firewalls useful? A pentester's view.

After doing some research on what I thought was a hot topic for the last few weeks - web application firewalls (WAFs) - I was surprised about what I saw at a recent vendor sponsored security conference: No mention of WAFs whatsoever! WAFs have been superseded by the Next Big Thing, which is "Next Generation Firewalls" (basically a revamped IPS/IDS that also includes a WAF).

It was also claimed at this conference that, based on a recent study, old-school packet filter firewalls block only 10% of attacks. This sounded about right. However it was also claimed that the remaining 90% would have been blocked if only a Next Generation Firewall had been in place. Web application attacks were given as an example of the kind of attacks being prevented.

If we assume that Next Generation Firewalls work like existing WAFs, this statement is simply not true. In fact, our experience with WAFs so far has shown quite the opposite: If there is a serious vulnerability in a web application, it doesn't matter if you put a WAF in front of it or not. Even if there is a WAF, a skilled attacker can almost always still exploit the application. In fact, because a WAF introduces an additional piece of software into your setup, it may even create additional possibilities for attack.

The main problem is: A WAF has to make extremely difficult decisions. A firewall is not a magic wall that somehow blocks attackers. It's a simple administrative tool. Traditional packet filter firewalls are so effective because they "just work". The filtering is done based on properties of the TCP/IP packet, so there's just a few things you can base your filter rules on. It's straightforward to configure and almost impossible to mess up.

Here is an iptables rule:

iptables -A INPUT -p tcp -s 0/0 -d 0/0 --destination-port 80--syn -j ACCEPT

This allows inbound HTTP connections. Let's compare this to at a typical rule from ModSecurity:

SecRule REQUEST_COOKIES|REQUEST_COOKIES_NAMES|REQUEST_FILENAME|ARGS_NAMES|ARGS|XML:/* "(?i:(?i:\d[\"'`´’‘]\s+[\"'`´’‘]\s+\d)|(?:^admin\s*?[\"'`´’‘]|(\/\*)+[\"'`´’‘]+\s?(?:--|#|\/\*|{)?)|(?:[\"'`´’‘]\s*?(x?or|div|like|between|and)[\w\s-]+\s*?[+<>=(),-]\s*?[\d\"'`´’‘])|(?:[\"'`´’‘]\s*?[^\w\s]?=\s*?[\"'`´’‘])|(?:[\"'`´’‘]\W*?[+=]+\W*?[\"'`´’‘])|(?:[\"'`´’‘]\s*?[!=|][\d\s!=+-]+.*?[\"'`´’‘(].*?$)|(?:[\"'`´’‘]\s*?[!=|][\d\s!=]+.*?\d+$)|(?:[\"'`´’‘]\s*?like\W+[\w\"'`´’‘(])|(?:\sis\s*?0\W)|(?:where\s[\s\w\.,-]+\s=)|(?:[\"'`´’‘][<>~]+[\"'`´’‘]))" 

What does this rule do? Right, you have no idea, because nobody can actually read regular expressions like this! And this is part of the first problem with WAFs. The rules required to detect or block any form of attack are extremely complex, if its even possible at all.

Same as with a packet filter firewall, the WAF is based on rules. But to define the right rules, you have to know in advance what an attack on the web application you are going to protect might look like. To know that, you need to know exactly how the web application works and what kinds of vulnerabilities it has. And even if you knew exactly how your web application was going to be attacked, it would be extremely difficult to define the right rules to prevent the attack. It is usually more difficult than fixing the application itself.

The following is a list of some of the main problems with WAFs.

WAFs only work with standard architectures 


Web application firewalls work well with standard web applications, such as simple PHP or Java Servlet applications. In the real world you have some of these standard web applications, but web applications can also look and behave very strangely. Just look at the kind of requests that Jetspeed and other Java Frameworks generate - without reading the source code of the web application you have no idea what's going on - and neither has a WAF. I have seen people developing their own Java scripting engines, using obscure programming languages (anybody heard of Erlang?) and even coding their own web servers. You cannot expect WAFs to deal with architectures like that.

WAFs only block standard attacks


WAFs can detect attacks only if they look like attacks are expected to look like. So usually that would be Cross-Site-Scripting, SQL injection, and so on that are exploited via the usual input channels. But there are many vulnerabilities that do NOT fit into these patterns. This kind of vulnerabilities is actually quite common.

As an example, a web application recently had a vulnerability where it was possible to upload ZIP archives which were then unpacked on the web server. The unpacking was done by some Java code that scanned the filenames listed in the archive and then unpacked the files into some target directory on the server. The problem here was that there was no sanity check on the filenames contained in the ZIP archive, so it was possible to extract a file directly into the web root of the application and by doing this, execute arbitrary code on the server.

Now if there was a typical WAF in place would it detect this kind of attack? Only if it was actually scanning the content of ZIP files that were uploaded. In theory it would be possible for a WAF to parse and inspect an uploaded ZIP archive. But then there would be a lot of other filetypes that would have to be inspected as well. Anyway, no WAF that I know of does this.

Restrictive rulesets generate too many false positives


If you actually want to prevent a large amount of attacks with a WAF you have to use a ruleset that is extremely restrictive. A good example for this is the OWASP core ruleset for mod_security. If you use this ruleset in the default configuration, it will block nearly everything that even remotely looks like an attack. If you have a ruleset like this, it indeed becomes more difficult to perform some kinds of attacks. But on the other hand, in most applications that have a lot of user interaction you will get a lot of false positives.

Here are some examples for things that would cause a false alarm with mod_security. Any of these sentences in user input would be blocked by the OWASP base rules.

today i selected orange juice from the juice machine
i am feeling well today :) and you?
that’s a nice curl you got there in your hair

WAFs do not protect against application logic attacks


There are many forms of attacks that WAFs do not detect at all. This includes all kinds of logical vulnerabilities. Imagine the following HTML comment on an admin login page:

<!-- The admin password is vnjr83rh43!, don't forget to delete this comment later! --!>

Somebody put the administrator password into the HTML source and you could simply use it to login the the admin interface and compromise the application. Now for a WAF, this would not look like an attack - the WAF has no way of knowing if the person logging in is a legitimate administrator or an attacker.

For a human attacker it is easy to apply the facts he learns about the web application. Sometimes we find trivial issues that cannot be detected by any kind of automated scanner of WAF.

Other logical issues include missing or insufficient authorization and issues with session management. Additionally, as the WAF does not actually know or understand the business rules of the application, it cannot detect possible violations of these rules.

There are many simple ways to bypass WAF rulesets


Even for the kind of attacks that should be prevented, there's an almost unlimited number of ways to bypass the WAF. SQL injection attacks are a good example. WAFs have a number of patterns that are unique to SQL injection attacks and check the supplied content for these patterns.

Let's say you find a simple SQL injection vulnerability and want to exploit it. Your goal will be to find out what exact pattern the WAF uses to detect this kind of attack and then obfuscate the attack so it does not match the pattern anymore. Here are some simple examples for rewriting a UNION SELECT statement that I have actually used to bypass WAFs.

p1 = 1+UNION++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++SELECT+cc+FROM+creditcards+--

?p1 = 1+UNION%09SELECT+cc+FROM+creditcards+--

?p1 = 1+UNION%0aSELECT+cc+FROM+creditcards+--

?p1 =  UNION\s+SELECT?p1=1+UNION/**/SELECT+cc+FROM+creditcards --

Attackers will study the database manual to find different ways to make the database query they want. They know that the WAF has a limited set of patterns that it can detect and in most cases it will be possible to find some way to make a query that is not detected.

WAF parsers are easily confused


Sometimes a WAF has such a restrictive ruleset that it is just not possible to get trough with an attack. The current version of mod_security with OWASP CRS is a good example for this, which is very restrictive even in the default configuration. In this case, the attacker has to rely on other techniques to bypass it. In that case, there is a whole other class of bypass techniques that rely on the following principle.

In a WAF setup every request that is received is parsed twice. First, it is parsed by the WAF to determine if there is an attack or not. If the WAF thinks that the request is legit, it passes it on to the web server for processing. Now in many cases, the WAF may interpret parts of the request in a different way from how it is then processed by the web application. And if we know exactly how, we will be able to evade detection by the WAF. Some also call this the "impedance mismatch" issue.

One example for this is HTTP parameter pollution. This sounds very fancy, but it simply means passing the same parameter to the server multiple times [1].

Another area that is traditionally problematic is multipart requests. A multipart request is the kind of request that is sent when you upload a file, for instance. Still, you can send every POST-request with a multipart encoding and this is processed like any normal POST request by the web server.

Parsing multipart requests is rather tricky and WAFs have the problem of having to match every parser by every web server and scripting language out there. It is often possible to bypass detection by a WAF by sending multipart requests that are interpreted differently by the WAF.

ModSecurity for instance has a very strict multipart parser to prevent this kind of attack. This means that it sets very strict limits for what kind of multipart requests it lets through. Even so, there are ways to bypass it. For example, a way to evade detection using single quotes in multipart requests was documented by Stefan Esser in 2009 [2].

Another bypass I found last month also has to do with parsing multipart requests. This time we use a double Content-Disposition header and terminate the first header with an additional carriage return. Again, this allows bypass of ModSecurity for arbitrary POST parameters [3].

Encodings are another problem area. My colleague Gerhard recently found a bypass technique for a commercial WAF called Airlock. In this case it was possible to completely bypass the WAF by supplying Null-bytes with an overlong UTF-8 encoding [4].

So even if you have a WAF with restrictive rules then it is usually possible to bypass it by using different forms of requests, encodings, and so on until something is found that the WAF parser does not understand but the web server does.

WAFs may introduce additional vulnerabilities


By installing a WAF you introduce an additional piece of software into your network and this software may have vulnerabilities of its own as well. A WAF has its own HTTP parser, for example. If this is not developed securely, then there will be buffer overflows, format string vulnerabilities and so on. So by turning on the WAF you might actually give an attacker additional opportunities to hack your server.

Commercial WAFs are usually not tested that much by security researchers as they are not that widely used and it is a bit tedious to obtain trial versions - you usually have to contact a sales rep. So it might be easier to find vulnerabilities in these WAFs than in well-tested software such as Apache and PHP.

During my latest research I have been looking at a commercial WAF and found a format string vulnerability that allowed execution by of arbitrary code on the affected system. Using this WAF would possibly allow an attacker to compromise the web server, even if the web application itself would have been secure (this vulnerability has not been made public yet).

Conclusion


While WAFs sound very good in theory they do not perform all that well in practice. In fact there are many problems with them. First, there are many types of vulnerabilities that WAFs do not protect against at all. And for those that they should protect against there are usually ways to bypass this protection.

The only scenario where WAFs should be used is as a workaround. If you know that a web application  you are running is vulnerable and you have really no way of fixing the application itself then you can use a WAF, but you have to remember that the default configuration will not be enough to protect the application. You need to know exactly which vulnerabilities the application has and you will have to configure the WAF to block exactly these attacks.

And finally, before you buy a WAF, make sure that it does not introduce new vulnerabilities into your system. Only use a WAF if you can be sure that it has been developed with secure software development practices in mind. Ask the WAF vendor what processes he has in place to ensure this.

References


[1]  http://www.iseclab.org/people/embyte/slides/BHEU2011/whitepaper-bhEU2011.pdf (Paper on HPP by Marco Balduzzi)
[2] http://www.suspekt.org/downloads/RSS09-WebApplicationFirewallBypassesAndPHPExploits.pdf (Stefan Esser's WAF Evasion)
[3] http://seclists.org/fulldisclosure/2012/Oct/113
[4] http://seclists.org/fulldisclosure/2012/Jun/310