Wednesday, November 25, 2015

House of Keys: Industry-Wide HTTPS Certificate and SSH Key Reuse Endangers Millions of Devices Worldwide

© seanlockephotography | Fotolia

In the course of an internal research project we have analyzed the firmware images of more than 4000 embedded devices of over 70 vendors. The devices we have looked at include Internet gateways, routers, modems, IP cameras, VoIP phones, etc. We have specifically analyzed cryptographic keys (public keys, private keys, certificates) in firmware images. The most common use of these static keys is:
  • SSH Host keys (keys required for operating a SSH server)
  • X.509 Certificates used for HTTPS (default server certificate for web based management)
In total we have found more than 580 unique private keys distributed over all the analysed devices. Correlation via the modulus allows us to find matching certificates.

We have correlated our data with data from Internet-wide scans ( and and found that our data set (580 unique keys) contains:
  • the private keys for more than 9% of all HTTPS hosts on the web (~150 server certificates, used by 3.2 million hosts)
  • the private keys for more than 6% of all SSH hosts on the web (~80 SSH host keys used by 0.9 million hosts)
So in total at least 230 out of 580 keys are actively used. Other research has pointed out the extent of this problem (Heninger, Nadia, et al. "Mining Your Ps and Qs: Detection of Widespread Weak Keys in Network Devices", Durumeric, Zakir, et al. "Analysis of the HTTPS certificate ecosystem"). However using our approach, an attribution at a vendor/product level is now possible. Plus the private keys have now been obtained.

Focus of analysis by SEC Consult

How do static keys come into existence?

They have been embedded, essentially "baked in" the firmware image (operating system) of devices and are mostly used for providing HTTPS and SSH access to the device. This is a problem because all devices that use the firmware use the exact same keys. The impact is described later in this post. The keys are not a result of bad randomness (e.g. Debian weak keys, PRNG boot gap, etc.).

The source of the keys is an interesting aspect. Some keys are only found in one product or several products in the same product line. In other cases we found the same keys in products from various different vendors. The reasons vary from shared/leaked/stolen code, white-label devices produced by different vendors (OEM, ODM products) to hardware/chipset/SoC vendor software development kits (SDKs) or board support packages firmware is based on. Here are some examples:
  • A certificate issued to a "Daniel", email ( is used in firmware from Actiontec, Aztech, Comtrend, Innatech, Linksys, Smart RG, Zhone and ZyXEL. This certificate is found in a Broadcom SDK. The affected vendors used it as a basis to develop their own firmware. More than 480.000 devices on the web are using this single certificate.
  • A certificate issued to Multitech in Bangalore, India is used in firmware from Aztech, Bewan, Observa Telecom, NetComm Wireless, Zhone, ZTE and ZyXEL. This certificate can be attributed to a Texas Instruments SDK for ADSL2+ routers. Over 300.000 devices on the web are using this certificate. A SSH host key can be attributed to this SDK as well.
  • A certificate issued to "MatrixSSL Sample Server Cert" is used in WiMAX gateways from Green Packet, Huawei, Seowon Intech, ZTE and ZyXEL. All affected devices use the same code base, which is likely developed by ZyXEL. At least 80.000 devices on the web are using this certificate.

Why are so many devices exposed to the web?

Another aspect to the whole story is the large number of devices directly accessible on the web. Just by looking at the numbers one can deduce that it is highly unlikely that each device is intentionally exposed on the web (remote management via HTTPS/SSH from WAN IP). Enabling remote management exposes an additional attack surface and enables attackers to exploit vulnerabilities in the device firmware as well as weak credentials set by the user.

In some cases this behaviour can be attributed to a vendor's insecure default configuration. An example is Ubiquiti Networks, who have remote management enabled by default in most products. This issue is documented in a separate blog post. We have also noticed that there is a surprisingly high number of Seagate devices on the web. Around 80.000 Seagate GoFlex home NAS devices are exposing HTTPS and SSH. The Seagate Share feature sets up port forwarding via UPnP. This feature is most likely responsible for a lot of the exposed devices and it is unclear if users are even aware of this problem.

In other cases ISPs expose their subscriber's devices (CPE - customer premises equipment) to the Internet. By correlating the affected hosts with GeoIP information, we found large clusters of devices with the same keys located in the networks of different ISPs. We can deduce that devices are CPEs provided to subscribers. These devices are owned, distributed and managed by ISPs and use ISP-specific firmware. Some ISPs have a particularly bad track record when it comes to exposing remote management interfaces:
  • US-based ISP CenturyLink exposes HTTPS remote administration on more than half a million devices, which is close to 10 percent of their total subscribers (6.1 million). Affected products include ZyXEL's Q1000, C1000Z, Actiontec's GT784WN, C2000A, C1000A, V1000H, and Technicolor's C2000T and C2100T.
  • TELMEX in Mexico exposes HTTPS remote administration on more than one million devices (22 million subscribers total). Affected products are various Huawei Internet Gateways including HG658d.
  • Telefónica in Spain (Movistar) exposes SSH remote administration on more than 170.000 devices. Affected products are Comtrend's VG-8050 and possibly some ZyXEL DSL gateways.
  • China Telecom exposes SSH remote administration on more than 100.000 ZyXEL and ZTE devices.
  • VTR Globalcom in Chile exposes SSH remote administration on more than 55.000 Ubee Interactive cable modems.
  • Chunghwa Telecom in Taiwan exposes SSH remote administration on more than 45.000 ZyXEL devices.
  • Telstra in Australia exposes SSH remote administration on more than 26.000 Cisco devices.
Top 10 Countries

What is the impact of the vulnerability?

Impersonation, man-in-the-middle or passive decryption attacks are possible. These attacks allow an attacker to gain access to sensitive information like administrator credentials which can be used in further attacks. In order to exploit this vulnerability, an attacker has to be in the position to monitor/intercept communication. This is easily feasible when the attacker is located within the same network segment (local network). Exploiting this vulnerability via the Internet is significantly more difficult, as an attacker has to be able to get access to the data that is exchanged. Attack vectors can be BGP hijacking, an "evil ISP", or a global adversary with the capability to monitor Internet traffic.

Searching for key fingerprints in data from Internet-wide scans is a low-cost way of finding the IP addresses of specific products/product groups. This enables researchers to measure the extent of the problem but attackers can use this approach as well to exploit vulnerabilities (e.g. weak passwords or vulnerabilities in firmware) at scale.

Which vendors/products are affected?

We found more than 900 products from about 50 vendors to be vulnerable. Of course our data is limited to the firmware we had access to. Affected vendors are: ADB, AMX, Actiontec, Adtran, Alcatel-Lucent, Alpha Networks, Aruba Networks, Aztech, Bewan, Busch-Jaeger, CTC Union, Cisco, Clear, Comtrend, D-Link, Deutsche Telekom, DrayTek, Edimax, General Electric (GE), Green Packet, Huawei, Infomark, Innatech, Linksys, Motorola, Moxa, NETGEAR, NetComm Wireless, ONT, Observa Telecom, Opengear, Pace, Philips, Pirelli , Robustel, Sagemcom, Seagate, Seowon Intech, Sierra Wireless, Smart RG, TP-LINK, TRENDnet, Technicolor, Tenda, Totolink, unify, UPVEL, Ubee Interactive, Ubiquiti Networks, Vodafone, Western Digital, ZTE, Zhone and ZyXEL.

We have generated tree maps in order to interactively explore the HTTPS certificate and SSH host key exposure on the public web including distribution on a per-country and ISP basis. Affected products are included as well in the hover boxes.

SSH Host Keys

Did you inform affected parties?

We have worked together with CERT/CC to address this issue since early August 2015. CERT/CC made great efforts to inform affected device vendors, chipset manufacturers and affected ISPs. Some vendors have responded and have started to implement fixes. CERT/CC informed major browser vendors as well. This vulnerability is tracked as VU#566724. Several CVE-IDs have been assigned as well.

What can be done?

As a first step, vendors should make sure that each device uses random, unique cryptographic keys. These can be computed in the factory or on first boot. In the case of CPE devices, both the ISP and the vendor have to work together to provide fixed firmware for affected devices.

Furthermore ISPs have to make sure remote access via the WAN port to CPEs is not possible. In case the ISP needs access for remote support purposes, setting up a dedicated management VLAN with strict ACLs (no CPE to CPE communication) is recommended.

End users should change the SSH host keys and X.509 certificates to device-specific ones. This is not always possible as some products do not allow this configuration to be changed or users do not have permissions to do it (frequent in CPE devices). The required technical steps (generating a certificate or RSA/DSA key pair etc.) are not something that can be expected of a regular home user.

Recommended selected elements of a mitigation strategy for vendors to avoid deploying insecure, ‘toxic’ devices are:
  • Execution of application security tests with high assurance level for all product components
  • Enforcement of Software Security Assurance in the whole software development process
  • Ongoing reevaluation of the maturity of application security
  • Maintenance of trust in the market through a long-term, fundamental approach to application security rather than a reactive and opportunistic one

All identified certificates and private keys will be released shortly.

We want to thank CERT/CC for coordinating this vulnerability and the team at the University of Michigan and Rapid7 for publishing their Internet-wide scan data.

This study was conducted by Stefan Viehböck, Senior Security Consultant at SEC Consult.

Thursday, November 5, 2015

The Omnipresence of Ubiquiti Networks Devices on the Public Web

There are ongoing in the wild attacks against Ubiquiti Networks devices. Attackers are using default credentials to gain access to the affected devices via SSH. The devices are infected by a botnet client that is able to infect other devices.

Further information about these attacks is available at:

The high number of Ubiquiti Networks devices on the web is due to an insecure default configuration. Remote administration is enabled by default. Contrary to best practice this exposes the HTTP/HTTPS and SSH administration service to the web (via the WAN port). This issue has been discussed on the support forums on several occasions. The Ubiquiti Networks employees' statements were:

"After we disabled management to WAN interface by default, there were many many complaints. Many WISPs still need access to the WAN interface for: provisioning the devices, and management -- even when in place at customer's home, the WISPs still want access." -Ubiquiti Networks Support

"We did disable remote management by default, and took a lot of flack from our users, so we reverted it." -Ubiquiti Networks Support

This "insecure out of the box" approach is unique among network device vendors. WISPs are Wireless Internet Providers, who offer broadband services using Ubiquiti Networks gear. Ars Technica has a feature on one of these WISPs.

Attackers can remotely exploit weak passwords and vulnerabilities in the firmware to gain access to affected devices and are doing just that right now. After gaining access to the device, attackers can create a large botnet that can be used for DDoS attacks, spamming etc. The affected device class is a valuable target as it not only allows access to the internal network but also access to traffic that passes through it. Attackers can redirect DNS queries (e.g. to attack online banking sites) or even infect web downloads by people who connect to the internet via the device.

As organizations and individuals struggle with securing embedded devices, a long lifespan of an infection is guaranteed.

Another vulnerability lies in the reuse of cryptographic secrets. A certificate including its private key is embedded in the firmware of several Ubiquiti Networks products. This certificate is used for the HTTPS service (default server certificate for web based management) and is the same on all devices. An attacker in a privileged network position can execute man-in-the-middle or passive decryption attacks on the HTTPS communication. These attacks allow access to sensitive information like admin credentials. This vulnerability is significantly harder to exploit than the vulnerability discussed above. However, searching for the certificate fingerprint in data from internet-wide scans is a low-cost way of finding the IPs of specific products/product groups and allows us to measure how many Ubiquiti Networks devices are exposed to the web.

The project by University of Michigan and Rapid7 periodically scans the public internet and collects information. By correlating the fingerprint of the certificate used with the Sonar SSL Certificates data set, we can find devices that are intentionally or unintentionally exposed to the web. In a recent scan we found more than 600.000 devices. We have analyzed the distribution of other static cryptographic secrets in use in embedded devices and have yet to find a certificate that is more frequently used than one by Ubiquiti Networks devices.

In the meantime the Censys Project by University of Michigan has launched. Censys is a scan-driven search engine that allows us to quickly find hosts that use the certificate. At the moment Censys reports that more than 1.100.000 devices are using this certificate (eb54c44a32a64497d8926ff87ba708f96fb0bff3).

Most devices are located in Brazil (480.000), Thailand (170.000) and the United States (77.000), but deployments of significant size are found in the Iraq, Spain and Poland as well. We have created an interactive tree map that allows to explore the geographical distribution of affected hosts at a country and ISP/organization level. The tree map is based on the data and covers less hosts than the data by Censys.

We have contacted Ubiquiti Networks back in August and communicated our findings (further information can be found our security advisory). While they have acknowledged the reuse of cryptographic secrets and proposed a fix, they have yet to comment on the remote administration issue.

Users can protect themselves from attacks by disabling remote administration and setting a strong administrator password. Changing the certificate to a custom one is also highly recommended as well.

The advisory contains technical information found in the certificate including corresponding RSA private key. Another certificate (fingerprint 61c96773fa2064148511d5cae262b721ba001fb6) was found to be used by other Ubiquiti Networks products as well, and is used on around 14.000 devices on the web.

Reuse of cryptographic secrets in HTTPS server certificates as well as SSH (host keys) is a widespread issue in embedded systems. We are in the process of preparing a larger study that will be released later this month.

Update: Some users have requested a list of affected products. We have documented the results of our analysis (incl. product list) in our security advisory. The list is not necessarily complete though and has not been confirmed by Ubiquiti Networks.

Wednesday, November 4, 2015

Interview: SEC Consult Praktikant Thomas Weber über die gewonnene European Cyber Security Challenge

Thomas Weber, former SEC Consult intern, was part of the winning team of the European Cyber Security Challenge (ECSC). We congratulate him and the team for this great success! It was a good opportunity for him to apply the knowledge he gained in his study at the Technical University of Vienna (Master Embededd Systems), during his internship at SEC Consult and through additional self motivated studies. He looks forward to participate in next years ECSC.

The complete interview with Thomas is available in German:


Die zweite European Cyber Security Challenge (ECSC) ist geschlagen! Beim Finale am 22. Oktober in Bern setzte sich das österreichische Nationalteam gegen Deutschland, Rumänien, Spanien, Großbritannien und die Schweiz durch. Unter den Siegern ist Thomas Weber, 25 Jahre, Student an der Technischen Universität Wien und ehemaliger Praktikant bei SEC Consult. Wie er die European Cyber Security Challenge erlebt hat und warum seine Praktikumserfahrung maßgeblich zum Sieg beitrug, erzählt er im Interview im SEC Consult Blog.

Credits: Swiss Cyber Storm

Erstmal herzliche Gratulation zum Sieg! Wie fühlt man sich, gegen so starke Teams angetreten zu sein und doch gewonnen zu haben?
Ich bin einfach stolz auf unser Team. Wir haben gezeigt, dass wir uns auf einem hohen Niveau behaupten können und das ist schon ein gutes Gefühl.

Wie kam es überhaupt zu deiner Teilnahme an der European Cyber Security Challenge?
Ich wurde durch einen Online-Artikel auf diesen Wettbewerb aufmerksam. Da ich gerne Rätsel löse und mich auch im Zuge meines Studiums (Master Embededd Systems) viel mit Computer beschäftige, habe ich mich angemeldet.

Das Team Österreich musste ja zuerst formiert werden, oder?
Ja, die nationale Ausscheidung begann mit einer Online-Qualifikation, bei der sich jeweils zehn Schüler und zehn Studenten qualifizieren konnten. Dabei mussten einzelne Aufgaben gelöst werden, wofür es Punkte gab. Wer im Gesamtranking bei den ersten zehn dabei war, kam ins österreichische Finale. Bei diesem wurden wir in Gruppen zu fünf Leuten gelost und hatten einen gewissen Zeitrahmen, in dem wieder Aufgaben gelöst werden mussten – natürlich vor dem konkurrierenden Team, da die Gruppe mit der schnelleren (und richtigen) Abgabe mehr Punkte bekam.

Wie hast du dich (bzw. auch das gesamte Team) auf das Finale in der Schweiz vorbereitet?
Wir haben begonnen, gezielt Strategien vorzubereiten, um optimal auf diese gemischte Form des Jeopardy/Attack-Defense CTFs eingestellt zu sein. Es haben sich nach und nach Leute im Team gefunden, die die verschiedenen Rollen (Server-Infrastruktur, Attack/Defense, Pressearbeit...) übernahmen und sich das entsprechende Wissen aneigneten. Wir haben uns diverse Programmiersprachen näher angesehen und mit der in der Schweiz zu erwartenden Infrastruktur verschiedene Szenarien durchprobiert, da wir einen ESXi Server von der CSA (Cyber Security Austria) zur Verfügung gestellt bekamen. Um das Netzwerk mussten wir uns nicht selbst kümmern, das übernahm die Organisation der Swiss Cyber Storm, die den Bewerb organisierte.

Was war deine Rolle / dein Part im Team Österreich?
Ich war einer der Leute, die Webapplikationen auf Schwachstellen untersuchten, um diese dann bei den anderen Teams ausnützen zu können. Jedes Team  hatte auf den Teamservern die gleichen Dienste am Laufen.

Was waren die Aufgabenstellungen?
Es gab mehrere Reverse-Engineering Aufgaben, die auf bereitgestellten Laptops zu lösen waren. Dabei musste auch der Netzwerkverkehr analysiert und ein Custom-Algorithmus zur Verschleierung des Netzwerk-Traffics geknackt werden. Die üblichen CTF Webservices liefen auf bereitgestellten Teamservern, die es zu verteidigen galt. Dabei war eine Website mit absichtlich eingebauten Sicherheitslücken zu untersuchen und zu reparieren, sodass die anderen Teams diese Lücke nicht ausnützen konnten.

Was war die größte Herausforderung?
Die größte Herausforderung war der komplett neue Modus. Im Punktesystem sowie bei den Aufgaben gab es leider noch Pannen, die natürlich bei solch einem Wettbewerb passieren können – gerade, wenn er neu ist. Dies betraf unser Team am Anfang sehr stark, da unsere Attack-Punkte über einen längeren Zeitraum nicht gezählt wurden. Dadurch hatten wir ein starkes Defizit zu den anderen Ländern. Das war möglich, da einige Teams alle Services stoppten und kaum Punkte verloren. Dieser Punktefehler wurde jedoch nach zwei Stunden behoben, der erlittene Punkterückstand blieb allerdings. Ich bin jedoch zuversichtlich, dass die ECSC nächstes Mal viel reibungsloser ablaufen wird, nicht zuletzt weil die Veranstalter ein hervorragendes Spiel auf die Beine gestellt haben.

Wie konnten dir deine gesammelten Erfahrungen vom Praktikum bei SEC Consult im Rahmen der ECSC helfen?
Ich bin um einiges schneller und effizienter geworden, wenn es um das Erkennen der relevanten und irrelevanten Sicherheitslücken geht. Als ich bei SEC Consult begonnen habe, haben mir die Schulungen und Meetings in den ersten drei Wochen meines Praktikums sehr viel gebracht. Das schnelle Erkennen war auch beim ECSC gefordert, da man ansonsten bei unwichtigen Teilen eines Webdienstes viel Zeit
verlieren kann.

Was empfiehlst du anderen jungen Menschen, die ebenfalls White-Hat-Hacker werden wollen?
Das Zeug zum White-Hat hat jeder, der ausdauernd, neugierig und technisch interessiert ist. Empfehlenswert, aber nicht notwendig, ist ein Studium mit Informatikschwerpunkt. Allerdings kommt man nicht drum herum, sich selbst Wissen anzueignen. Derartige Wettbewerbe sind eine gute Möglichkeit viele Dinge durch Eigeninitiative zu lernen, daher lohnt es sich auf jeden Fall mitzumachen!

Wirst du nächstes Jahr wieder an der Cyber Security Challenge teilnehmen?

Wie sehen deine weiteren (Karriere-)Schritte aus?
Ich werde mein Masterstudium weiterführen und nach dem Abschluss in die Privatwirtschaft gehen.

Zur Person
Thomas Weber ist Student an der Technischen Universität Wien (Master Embededd Systems) im dritten Semester.  Zum Studium führte den 25-Jährigen seine Leidenschaft für Informatik, insbesondere im Bereich Hacking. Als professioneller White-Hat-Hacker möchte Weber in Zukunft das World Wide Web sicherer machen.