In the course of an internal research project we have analyzed the firmware images of more than 4000 embedded devices of over 70 vendors. The devices we have looked at include Internet gateways, routers, modems, IP cameras, VoIP phones, etc. We have specifically analyzed cryptographic keys (public keys, private keys, certificates) in firmware images. The most common use of these static keys is:
- SSH Host keys (keys required for operating a SSH server)
- X.509 Certificates used for HTTPS (default server certificate for web based management)
In total we have found more than 580 unique private keys distributed over all the analysed devices.
Correlation via the modulus allows us to find matching certificates.
- the private keys for more than 9% of all HTTPS hosts on the web (~150 server certificates, used by 3.2 million hosts)
- the private keys for more than 6% of all SSH hosts on the web (~80 SSH host keys used by 0.9 million hosts)
So in total at least 230 out of 580 keys are actively used. Other research has pointed out the extent of this problem (Heninger, Nadia, et al. "Mining Your Ps and Qs: Detection of Widespread Weak Keys in Network Devices", Durumeric, Zakir, et al. "Analysis of the HTTPS certificate ecosystem"). However using our approach, an attribution at a vendor/product level is now possible. Plus the private keys have now been obtained.
How do static keys come into existence?
- A certificate issued to a "Daniel", email (email@example.com) is used in firmware from Actiontec, Aztech, Comtrend, Innatech, Linksys, Smart RG, Zhone and ZyXEL. This certificate is found in a Broadcom SDK. The affected vendors used it as a basis to develop their own firmware. More than 480.000 devices on the web are using this single certificate.
- A certificate issued to Multitech in Bangalore, India is used in firmware from Aztech, Bewan, Observa Telecom, NetComm Wireless, Zhone, ZTE and ZyXEL. This certificate can be attributed to a Texas Instruments SDK for ADSL2+ routers. Over 300.000 devices on the web are using this certificate. A SSH host key can be attributed to this SDK as well.
- A certificate issued to "MatrixSSL Sample Server Cert" is used in WiMAX gateways from Green Packet, Huawei, Seowon Intech, ZTE and ZyXEL. All affected devices use the same code base, which is likely developed by ZyXEL. At least 80.000 devices on the web are using this certificate.
Why are so many devices exposed to the web?
- US-based ISP CenturyLink exposes HTTPS remote administration on more than half a million devices, which is close to 10 percent of their total subscribers (6.1 million). Affected products include ZyXEL's Q1000, C1000Z, Actiontec's GT784WN, C2000A, C1000A, V1000H, and Technicolor's C2000T and C2100T.
- TELMEX in Mexico exposes HTTPS remote administration on more than one million devices (22 million subscribers total). Affected products are various Huawei Internet Gateways including HG658d.
- Telefónica in Spain (Movistar) exposes SSH remote administration on more than 170.000 devices. Affected products are Comtrend's VG-8050 and possibly some ZyXEL DSL gateways.
- China Telecom exposes SSH remote administration on more than 100.000 ZyXEL and ZTE devices.
- VTR Globalcom in Chile exposes SSH remote administration on more than 55.000 Ubee Interactive cable modems.
- Chunghwa Telecom in Taiwan exposes SSH remote administration on more than 45.000 ZyXEL devices.
- Telstra in Australia exposes SSH remote administration on more than 26.000 Cisco devices.
What is the impact of the vulnerability?
Impersonation, man-in-the-middle or passive decryption attacks are possible. These attacks allow an attacker to gain access to sensitive information like administrator credentials which can be used in further attacks. In order to exploit this vulnerability, an attacker has to be in the position to monitor/intercept communication. This is easily feasible when the attacker is located within the same network segment (local network). Exploiting this vulnerability via the Internet is significantly more difficult, as an attacker has to be able to get access to the data that is exchanged. Attack vectors can be BGP hijacking, an "evil ISP", or a global adversary with the capability to monitor Internet traffic.
Searching for key fingerprints in data from Internet-wide scans is a low-cost way of finding the IP addresses of specific products/product groups. This enables researchers to measure the extent of the problem but attackers can use this approach as well to exploit vulnerabilities (e.g. weak passwords or vulnerabilities in firmware) at scale.
We found more than 900 products from about 50 vendors to be vulnerable. Of course our data is limited to the firmware we had access to. Affected vendors are: ADB, AMX, Actiontec, Adtran, Alcatel-Lucent, Alpha Networks, Aruba Networks, Aztech, Bewan, Busch-Jaeger, CTC Union, Cisco, Clear, Comtrend, D-Link, Deutsche Telekom, DrayTek, Edimax, General Electric (GE), Green Packet, Huawei, Infomark, Innatech, Linksys, Motorola, Moxa, NETGEAR, NetComm Wireless, ONT, Observa Telecom, Opengear, Pace, Philips, Pirelli , Robustel, Sagemcom, Seagate, Seowon Intech, Sierra Wireless, Smart RG, TP-LINK, TRENDnet, Technicolor, Tenda, Totolink, unify, UPVEL, Ubee Interactive, Ubiquiti Networks, Vodafone, Western Digital, ZTE, Zhone and ZyXEL.
We have generated tree maps in order to interactively explore the HTTPS certificate and SSH host key exposure on the public web including distribution on a per-country and ISP basis. Affected products are included as well in the hover boxes.
Did you inform affected parties?
We have worked together with CERT/CC to address this issue since early August 2015. CERT/CC made great efforts to inform affected device vendors, chipset manufacturers and affected ISPs. Some vendors have responded and have started to implement fixes. CERT/CC informed major browser vendors as well. This vulnerability is tracked as VU#566724. Several CVE-IDs have been assigned as well.
What can be done?
As a first step, vendors should make sure that each device uses random, unique cryptographic keys. These can be computed in the factory or on first boot. In the case of CPE devices, both the ISP and the vendor have to work together to provide fixed firmware for affected devices.
Furthermore ISPs have to make sure remote access via the WAN port to CPEs is not possible. In case the ISP needs access for remote support purposes, setting up a dedicated management VLAN with strict ACLs (no CPE to CPE communication) is recommended.
End users should change the SSH host keys and X.509 certificates to device-specific ones. This is not always possible as some products do not allow this configuration to be changed or users do not have permissions to do it (frequent in CPE devices). The required technical steps (generating a certificate or RSA/DSA key pair etc.) are not something that can be expected of a regular home user.
Recommended selected elements of a mitigation strategy for vendors to avoid deploying insecure, ‘toxic’ devices are:
- Execution of application security tests with high assurance level for all product components
- Enforcement of Software Security Assurance in the whole software development process
- Ongoing reevaluation of the maturity of application security
- Maintenance of trust in the market through a long-term, fundamental approach to application security rather than a reactive and opportunistic one
All identified certificates and private keys will be released shortly.
We want to thank CERT/CC for coordinating this vulnerability and the Scans.io/ZMap/Censys team at the University of Michigan and Rapid7 for publishing their Internet-wide scan data.
This study was conducted by Stefan Viehböck, Senior Security Consultant at SEC Consult.