Security researchers recently found gaping vulnerabilities in a wide variety of critical business and industrial equipment. It turns out that weak or absent passwords made it easy to break into more than 100,000 terminal servers used to provide their Internet connections. Fixing the problem is simple. Change the credentials dramatically reduces the risk. But for many companies, actually solving the problem is nearly impossible.

Vulnerable, But Hidden The threats discovered by security firm Rapid7 exemplify the difficulties organizations face in plugging even known holes in critical gear. In this case, the affected systems include industrial control equipment, traffic-signal monitors, fuel pumps, retail point-of-sale terminals and building automation equipment such as alarms and heating and ventilation (HVAC) systems.

Rapid7 found more than 114,000 unprotected terminal servers, mostly from Digi International or Lantronix, that a hacker could use to take control of the underlying systems. Finding the serial ports on the server requires the use of a scanning tool, such as Nmap. Once an active port is found, a command-line program similar to what those used in 1980s vintage home computers is all that's needed to access a control panel or menu or capture data.

Fortunately, while tech-savvy saboteurs or terrorists would have no difficulty gaining access to the equipment, they most likely would not know who owns it or where it is located. Without that information, the find would not be very useful. "There's no telling who they are going to hurt, if they don't know where the device is," explained HD Moore, chief research officer for Rapid7.

How Security Gets Missed Nevertheless, any hole that can provide access to critical equipment is worth plugging, but it's not likely to happen in many of these cases. Often, companies do not even know the terminal server exists, much less that it needs security updates.

How is that possible? Well, picture a vendor working with the facilities crew installing an HVAC system that uses a terminal server so the equipment can be monitored from a remote location. No one knows the server exists, and no one cares, as long as everything works. "A lot of times IT is not even aware of these systems," said Matthew Neely, director of research at risk management company SecureState.

Vendor marketing can also exacerbate the problem. Equipment is often sold as being "secured," when in fact it is only "capable of being secured." That means the buyer still has to add the technology or turn on and configure the security features.

This can get missed if the installers assume the equipment is "plug and play," said Joe Weiss, a security consultant for Applied Control Solutions. "It's like getting a toy for Christmas and you pull it out of the box expecting it to run, because the box doesn't tell you it needs two AA batteries," Weiss added.

Terminal servers, also called serial port servers, often get missed by electric utility companies because they are not covered under federal cybersecurity requirements. So the devices never make it on the utility's compliance checklist. "They don't even have to check these out to find out if they are or not secure," Weiss said.

This bizarre situation demonstrates that ensuring the security of critical equipment is never a matter of technology alone. True security requires people to pay attention, not just sweep everything under the rug.

by Steve Lasky
Created: April 16, 2013

Pointing out the national magnitude of the event and the sheer volume of spectators the event brought to the downtown area, Wardell instituted educational symposiums for his staff in security and emergency preparedness that outlined basic response scenarios. “Bottom line, in large open events like this, is you need to understand who your partners are and what resources you have at your disposal,” Wardell says. “It is key to create a unified communications structure and command organization, and there has to be a real spirit of cooperation to make it work.”

Wardell believes the officials at the Boston Marathon did their due diligence and provided the most secure environment possible for fans and runners. “Unless you close off the entire venue, they did all they could do,” he says. “Securing a 26-mile course would be so manpower-intensive that it is just not feasible. You would have to bring in the military for such tactics. Now that is done for a Presidential move, but for an event like this, it would be overkill and much too cost-prohibitive.”

David Holley, a senior managing director for the Boston office of Kroll echoes the sentiment that events such as the Boston Marathon present huge security challenges. “Another difficulty in protecting an event like a marathon is the fact that there are no assigned seats and the spectator crowd is generally mobile,” he wrote in an article published The New York Daily News. “Moving from place to place to secure a better view, watch friends go by, or work their way down to the finish, the crowd is generally always in motion. Knapsacks and handbags are picked up and put down countless times, and frequently forgotten at the last location, making it difficult to determine whether an object has been abandoned or unintentionally left behind."

Indeed, the mess of “left behinds” after an event like a marathon is a daunting task to sift through and dispose of, he added. These things make it difficult to monitor and secure open-venue events, but it is also what makes them wonderful events to attend, he said.

Wardell agrees that having the ability to screen fans at specific ingress points is the most important difference between his Final Four and the marathon. “For the Final Four, even though we had open venues, they all had defined perimeters with access control — even if that access control was just staring at you and doing bag checks,” he says. “Fans and vendors (at the Final Four) were required to go through a gauntlet or checkpoint that allowed for screening. At the marathon, you had people standing 50-feet deep, sitting at cafes along the route. How do you prepare for that?