|© seanlockephotography | Fotolia|
In the course of an internal research project we have analyzed the firmware images of more than 4000 embedded devices of over 70 vendors. The devices we have looked at include Internet gateways, routers, modems, IP cameras, VoIP phones, etc. We have specifically analyzed cryptographic keys (public keys, private keys, certificates) in firmware images. The most common use of these static keys is:
- SSH Host keys (keys required for operating a SSH server)
- X.509 Certificates used for HTTPS (default server certificate for web based management)
We have correlated our data with data from Internet-wide scans (Scans.io and Censys.io) and found that our data set (580 unique keys) contains:
- the private keys for more than 9% of all HTTPS hosts on the web (~150 server certificates, used by 3.2 million hosts)
- the private keys for more than 6% of all SSH hosts on the web (~80 SSH host keys used by 0.9 million hosts)
How do static keys come into existence?
The source of the keys is an interesting aspect. Some keys are only found in one product or several products in the same product line. In other cases we found the same keys in products from various different vendors. The reasons vary from shared/leaked/stolen code, white-label devices produced by different vendors (OEM, ODM products) to hardware/chipset/SoC vendor software development kits (SDKs) or board support packages firmware is based on. Here are some examples:
- A certificate issued to a "Daniel", email (firstname.lastname@example.org) is used in firmware from Actiontec, Aztech, Comtrend, Innatech, Linksys, Smart RG, Zhone and ZyXEL. This certificate is found in a Broadcom SDK. The affected vendors used it as a basis to develop their own firmware. More than 480.000 devices on the web are using this single certificate.
- A certificate issued to Multitech in Bangalore, India is used in firmware from Aztech, Bewan, Observa Telecom, NetComm Wireless, Zhone, ZTE and ZyXEL. This certificate can be attributed to a Texas Instruments SDK for ADSL2+ routers. Over 300.000 devices on the web are using this certificate. A SSH host key can be attributed to this SDK as well.
- A certificate issued to "MatrixSSL Sample Server Cert" is used in WiMAX gateways from Green Packet, Huawei, Seowon Intech, ZTE and ZyXEL. All affected devices use the same code base, which is likely developed by ZyXEL. At least 80.000 devices on the web are using this certificate.
Why are so many devices exposed to the web?
In some cases this behaviour can be attributed to a vendor's insecure default configuration. An example is Ubiquiti Networks, who have remote management enabled by default in most products. This issue is documented in a separate blog post. We have also noticed that there is a surprisingly high number of Seagate devices on the web. Around 80.000 Seagate GoFlex home NAS devices are exposing HTTPS and SSH. The Seagate Share feature sets up port forwarding via UPnP. This feature is most likely responsible for a lot of the exposed devices and it is unclear if users are even aware of this problem.
In other cases ISPs expose their subscriber's devices (CPE - customer premises equipment) to the Internet. By correlating the affected hosts with GeoIP information, we found large clusters of devices with the same keys located in the networks of different ISPs. We can deduce that devices are CPEs provided to subscribers. These devices are owned, distributed and managed by ISPs and use ISP-specific firmware. Some ISPs have a particularly bad track record when it comes to exposing remote management interfaces:
- US-based ISP CenturyLink exposes HTTPS remote administration on more than half a million devices, which is close to 10 percent of their total subscribers (6.1 million). Affected products include ZyXEL's Q1000, C1000Z, Actiontec's GT784WN, C2000A, C1000A, V1000H, and Technicolor's C2000T and C2100T.
- TELMEX in Mexico exposes HTTPS remote administration on more than one million devices (22 million subscribers total). Affected products are various Huawei Internet Gateways including HG658d.
- Telefónica in Spain (Movistar) exposes SSH remote administration on more than 170.000 devices. Affected products are Comtrend's VG-8050 and possibly some ZyXEL DSL gateways.
- China Telecom exposes SSH remote administration on more than 100.000 ZyXEL and ZTE devices.
- VTR Globalcom in Chile exposes SSH remote administration on more than 55.000 Ubee Interactive cable modems.
- Chunghwa Telecom in Taiwan exposes SSH remote administration on more than 45.000 ZyXEL devices.
- Telstra in Australia exposes SSH remote administration on more than 26.000 Cisco devices.
What is the impact of the vulnerability?
Searching for key fingerprints in data from Internet-wide scans is a low-cost way of finding the IP addresses of specific products/product groups. This enables researchers to measure the extent of the problem but attackers can use this approach as well to exploit vulnerabilities (e.g. weak passwords or vulnerabilities in firmware) at scale.
Which vendors/products are affected?
We have generated tree maps in order to interactively explore the HTTPS certificate and SSH host key exposure on the public web including distribution on a per-country and ISP basis. Affected products are included as well in the hover boxes.
Did you inform affected parties?
What can be done?
Furthermore ISPs have to make sure remote access via the WAN port to CPEs is not possible. In case the ISP needs access for remote support purposes, setting up a dedicated management VLAN with strict ACLs (no CPE to CPE communication) is recommended.
End users should change the SSH host keys and X.509 certificates to device-specific ones. This is not always possible as some products do not allow this configuration to be changed or users do not have permissions to do it (frequent in CPE devices). The required technical steps (generating a certificate or RSA/DSA key pair etc.) are not something that can be expected of a regular home user.
Recommended selected elements of a mitigation strategy for vendors to avoid deploying insecure, ‘toxic’ devices are:
- Execution of application security tests with high assurance level for all product components
- Enforcement of Software Security Assurance in the whole software development process
- Ongoing reevaluation of the maturity of application security
- Maintenance of trust in the market through a long-term, fundamental approach to application security rather than a reactive and opportunistic one
We want to thank CERT/CC for coordinating this vulnerability and the Scans.io/ZMap/Censys team at the University of Michigan and Rapid7 for publishing their Internet-wide scan data.
This study was conducted by Stefan Viehböck, Senior Security Consultant at SEC Consult.