Server BMC Security Concerns – What They Don’t Know Can’t Hurt You

As anybody who has been a parent will tell you, children are voracious eaters (especially teenagers). Something my wife or I might buy as a component for a family dinner often becomes a single meal for one or more of our children as soon as it is discovered in the fridge or pantry. Having three children in the house makes it very difficult to identify the main culprit, but it is normally a combined effort from all of them as once the desirable item is discovered the children descend upon it like a pack of hyenas on a wildebeest carcass.

We have employed several strategies over the years. Ultimately teaching accountability and self-discipline won the day, but initially the most effective prevention was simply not leaving it out where it may be in their line of sight when they are browsing for a snack (hoping they will go for the fresh fruit sitting out on the counter instead). Reading the many articles available online about security vulnerabilities in IPMI and various vendor’s BMC (Baseboard Management Controller) implementations has caused me to reflect on this. Appealing to the humanity of hackers actively seeking to do harm seems like a losing strategy and much like children with the munchies they will always find a way. You may think making a management network accessible from the internet seems like an obviously terrible idea, but it is shockingly common. In 2014 a researcher was able to discover over 230,000 BMCs exposed to the internet by scanning for the IPMI protocol and as recently as July of 2020 Positive Technologies researchers found more than 500 BMC devices through public search engines.

https://securityledger.com/2014/06/ipmi-insecurity-affects-200k-systems/

There is a large variety of attack vectors for a general-purpose computer system, but the BMC is very specialized and purpose-built. The main vulnerabilities are the IPMI protocol, other management interfaces such as native CLI and web-based, and the Linux based kernel many BMCs are built upon. The BMC is a very high-value target as they provide an incredible level of access directly to the system hardware and can be used to disrupt operations, steal data, or even physically damage the hardware.

Some sort of embedded OS is part of the price of doing business and Linux is the obvious choice. Aside from maintaining security best practices and keeping it up to date there isn’t much more to be done; you can’t operate without it. Management and web interfaces are another necessary evil, although steps can be taken to significantly harden those. Dell’s iDRAC 9 for example, allows limiting access to certain IP addresses and an industry first system lockdown mode to prevent any system changes as a last line of defense. That brings us to the elephant in the room, the giant glowing neon “hack me” sign, none other than IPMI (intelligent platform management interface).

It is no secret to anybody in the industry that IPMI has had a troubled relationship with security. IPMI was the only game in town for a while but it still speaks volumes to the power and versatility of the protocol that it has been used ubiquitously for so many years despite the numerous, extremely dangerous, and well-documented security vulnerabilities. The CVE (Common Vulnerabilities and Exposures) list for IPMI reads like a hacker’s dream come true (https://cve.mitre.org/cgi-bin/cvekey.cgi?keyword=ipmi). To be fair, discovering exploits is to be expected but many of these skip the appetizer and go straight to dessert, allowing unrestricted access to the system in essentially one step. Documented issues allow things like clear-text authentication, allowing access with any password (Cipher 0), password hash retrieval, and authentication using a null user account. Most of these common issues are baked into IPMI 2.0 and enabled by default. (https://www.rapid7.com/blog/post/2013/07/02/a-penetration-testers-guide-to-ipmi/) This is all exacerbated by one huge cherry on top and that is that IPMI runs on its own port, UDP 623. A simple packet sniffer or portmapper could tell a hacker with network access where all the BMCs are located very easily. It is no wonder that there will be no further revisions of the IPMI standard, and the industry is moving to a much more secure and universal standard: Redfish.

From 2017 onward (the year of the first CVE for Redfish) there are 17 documented CVE entries for IPMI and a whopping 2 for Redfish. That may not be a fair comparison as Redfish (while widely supported) has yet to gain the same level of adoption, but it is still quite a delta. Redfish is a REST API built on the industry-standard OData and JSON formats making it relatively simple and cost-effective to adopt. It uses all standard HTTPS commands and ports, making it stand out much less in an environment, and benefits from the mature security of its parent protocol stack. I have several videos on this site talking about Redfish and its benefits, so we won’t go into further depth here but suffice it to say that disabling IPMI and using Redfish makes the iDRAC or any BMC significantly more secure.

Now we come back to where we started, security through obscurity. For the admins of those 500 BMCs found on the internet last year, or anyone who has never had their kids eat an entire package of a dozen donuts before he got one: If you don’t want people messing with it, then hide it well. If your network is secure enough, then IPMI would be just fine but where there is a will there is a way and all we can do is make it as difficult as possible every step of the way. Defense in-depth and diligent security practices are a deterrent, allow more time for detection, and with a little luck just maybe you can get that donut before they are all gone.