IoT Security for Developers [Survive IoT Part 5]

Previous articles focused on how to securely design and configure a system based on existing hardware, software, IoT Devices, and networks. If you are developing IoT devices, software, and systems, there is a lot more you can do to develop secure systems.

The first thing is to manage and secure communications with IoT Devices. Your software needs to be able to discover, configure, manage and communicate with IoT devices. By considering security implications when designing and implementing these functions you can make the system much more robust. The basic guideline is don’t trust any device. Have checks to verify that a device is what it claims to be, to verify device integrity, and to validate communications with the devices.

Have a special process for discovering and registering devices and restrict access to it. Do not automatically detect and register any device that pops up on the network! Have a mechanism for pairing devices with the gateway, such as a special pairing mode that must be invoked on both the device and the gateway to pair or a requirement to manually enter a device serial number or address into the gateway as part of the registration process. For industrial applications adding devices is a deliberate process – this is not a good operation to fully automate!

A solid approach to gateway and device identity is to have a certificate provisioned onto the device at the factory, by the system integrator, or at a central facility. It is even better if this certificate is backed by a HW root of trust that can’t be copied or spoofed.

Communications between the gateway and the device should be designed. Instead of a general network connection, which can be used for many purposes, consider using a specialized interface. Messaging interfaces are ideal for many IoT applications. Two of the most popular messaging interfaces are MQTT (Message Queued Telemetry Transport) and CoAP. In addition to their many other advantages, these messaging interfaces only carry IoT data, greatly reducing their capability to be used as an attack vector.

Message based interfaces are also a good approach for connecting the IoT Gateway to backend systems. An enterprise message bus like AMQP is a powerful tool for handling asynchronous inputs from thousands of gateways, routing them, and feeding the data into backend systems. A messaging system makes the total system more reliable, more robust, and more efficient – and makes it much easier to implement large scale systems! Messaging interfaces are ideal for handling exceptions – they allow you to simply send the exception as a regular message and have it properly processed and routed by business logic on the backend.

Messaging systems are also ideal for handling unreliable networks and heavy system loads. A messaging system will queue up messages until the network is available. If a sudden burst of activity causes the network and backend systems to be overloaded the messaging system will automatically queue up the messages and then release them for processing as resources become available. Messaging systems allow you to ensure reliable message delivery, which is critical for many applications. Best of all, messaging systems are easy for a programmer to use and do the hard work of building a robust communications capability for you.

No matter what type of interface you are using it is critical to sanitize your inputs. Never just pass through information from a device – instead, check it to make sure that is properly formatted, that it makes sense, that it does not contain a malicious payload, and that the data has not been corrupted. The overall integrity of an IoT system is greatly enhanced by ensuring the quality of the data it is operating on. Perhaps the best example of this is Little Bobby Tables from XKCD (XKCD.com):

Importance of sanitizing your input.

Importance of sanitizing your input.

On a more serious level, poor input sanitization is responsible for many security issues. Programmers should assume that users can’t be trusted and all interactions are a potential attack.

Posted in IoT, Security, System Management | Leave a comment

Security by Isolating Insecurity [Survive IoT Part 4]

In my previous post I introduced “Goldilocks Security”, proposing three approaches to security.

Solution 1: Ignore Security

Safety in the crowd – with tens of millions of cameras out there, why would anyone pick mine? Odds are that the bad guys won’t pick yours – they will pick all of them! Automated search and penetration tools easily find millions of IP cameras. You will be lost in the crowd – the crowd of bots!

Solution 2: Secure the Cameras

For home and small business customers, a secure the camera approach simply won’t work because ease of use wins out over effective security in product design and because the camera vendors’ business model (low-cost, ease of use, and access over the Internet) all conspire against security. What’s left?

Solution 3: Isolation

If the IP cameras can’t be safely placed on the Internet, then isolate them from the Internet.

To do this, introduce an IoT Gateway between the cameras and all other systems. This IoT Gateway would have two network interfaces: one network interface dedicated to the cameras and the second network interface used to connect to the outside world. An application running on the IoT Gateway would talk to the IP cameras and then talk to the outside world (if needed). There would be no network connection between the IP cameras and anything other than the IoT Gateway application. The IoT Gateway would also be hardened and actively managed for best security.

How is this implemented?

  • Put the IP cameras on a dedicated network. This should be a separate physical network. At a minimum it should be a VLAN (Virtual LAN). There will typically be a relatively small number of IP cameras in use, so a dedicated network switch, probably with PoE, is cost effective.
    • Use static IP addresses. If the IP cameras are assigned static IP addresses, there is no need to have an IP gateway or DNS server on the network segment. This further reduces the ability of the IP cameras to get out on the network. You lose the convenience of DHCP assigned address and gain significant security.
    • You can have multiple separate networks. For example, you might have one for external cameras, one for cameras in interior public spaces, one for manufacturing space and one for labs. With this configuration, someone gaining access to the exterior network would not be able to gain access to the lab cameras.
  • Add an IoT Gateway – a computer with a network interface connected to the camera network. In the example above, the gateway would have four network interfaces – one for each camera network. The IoT Gateway would probably also be connected to the corporate network; this would require a fifth network interface. Note that you can have multiple IoT Gateways, such as one for each camera network, one for a building management system, one for other security systems, and one that connects an entire building or campus to the Internet.
  • Use a video monitoring program such as ZoneMinder or a commercial program to receive, monitor and display the video data. Such a program can monitor multiple camera feeds, analyze the video feeds for things such as motion detection, record multiple video streams, and create events and alerts. These events and alerts can do things like trigger alarms, send emails, send text messages, or trigger other business rules. Note that the video monitoring program further isolates the cameras from the Internet – the cameras talk to the video monitoring program and the video monitoring program talks to the outside world.
  • Sandbox the video monitoring program using tools like SELinux and containers. These both protect the application and protect the rest of the system from the application – even if the application is compromised, it won’t be able to attack the rest of the system.
  • Remove any unneeded services from the IoT Gateway. This is a dedicated device performing a small set of tasks. There shouldn’t be any software on the system that is not needed to perform these tasks – no development tools, no extraneous programs, no unneeded services running.
  • Run the video monitoring program with minimal privileges. This program should not require root level access.
  • Configure strong firewall settings on the IoT Gateway. Only allow required communications. For example, only allow communications with specific IP addresses or mac addresses (the IP cameras configured into the system) over specific ports using specific protocols. You can also configure the firewall to only allow specific applications access to the network port. These settings would keep anything other than authorized cameras from accessing the gateway and keep the authorized cameras from talking to anything other than the video monitoring application. This approach also protects the cameras. Anyone attempting to attack the cameras from the Internet would need to penetrate the IoT Gateway and then change settings such as the firewall and SELinux before they could get to the cameras.
  • Use strong access controls. Multi-factor authentication is a really good idea. Of course you have a separate account for each user, and assign each user the minimum privilege they need to do their job. Most of the time you don’t need to be logged in to the system – most video monitoring applications can display on the lock screen, allowing visual monitoring of the video streams without being able to change the system. For remote gateways interactive access isn’t needed at all; they simply process sensor data and send it to a remote system.
  • Other systems should be able to verify the identity of the IoT Gateway. A common way to do this is to install a certificate on the gateway. Each gateway should have a unique certificate, which can be provided by systems like Linux IdM or MS Active Directory. Even greater security can be provided by placing the system identity into a hardware root of trust like a TPM (Trusted Processing Module), which prevents the identity from being copied, cloned, or spoofed.
  • Encrypted communications is always a good idea for security. Encryption protects the contents of the video stream from being revealed, prevents the contents of the video stream from being modified or spoofed, and verifies the integrity of the video stream – any modifications of the encrypted traffic, either deliberate or due to network error, are detected. Further, if you configure a VPN (Virtual Private Network) between the IoT Gateway and backend systems you can force all network traffic through the VPN, thus preventing network attacks against the IoT Gateway. For security systems it is good practice to encrypt all traffic, both internal and external.
  • Proactively manage the IoT Gateway. Regularly update it to get the latest security patches and bug fixes. Scan it regularly with tools like OpenSCAP to maintain secure configuration. Monitor logfiles for anomalies that might be related to security events, hardware issues, or software issues.

You can see how a properly configured IoT Gateway can allow you to use insecure IoT devices as part of a secure system. This approach isn’t perfect – the cameras should also be managed like the gateway – but it is a viable approach to building a reasonably secure and robust system out of insecure devices.

One issue is that the cameras are not protected from local attack. If WiFi is used the attacker only needs to be nearby. If Ethernet is used an attacker can add another device to the network. This is difficult as you would need to gain access to the network switch and find a live port on the proper network. Attacking the Ethernet cable leaves signs, including network glitches. Physically attacking a camera also leaves signs. All of this can be done, but is more challenging than a network based attack over the Internet and can be managed through physical security and good network monitoring. These are some of the reasons why I strongly prefer wired network connections over wireless network connections.

Posted in IoT, Security, System Management | Leave a comment

Goldilocks Security: Bad, Won’t Work, and Plausible [Survive IoT Part 3]

Previous posts discussed the security challenge presented by IoT devices, using IP Video Cameras as an example. Now let’s consider some security alternatives:

Solution 1: Ignore Security

This is the most common approach to IoT security today. And, to a significant degree, it works. In the same way that ignoring fire safety usually works – only a few businesses or homes burn down each year!

Like fire safety, the risks from ignoring IoT security grow over time. Like fire safety, the cost of the relatively rare events can be catastrophic. Unlike fire safety, an IoT event can affect millions of entities at the same time.

And, unlike traditional IT security issues, IoT security issues can result in physical damage and personal injury. Needless to say, I do not recommend ignoring the issue as a viable approach to IoT security!

Solution 2: Secure the Cameras

Yes, you should secure IP cameras. They are computers sitting on your network – and should be treated like computers on your network! Best practices for IT security are well known and readily available. You should install and configure them securely, update them regularly, and monitor them continuously.

If you have a commercial implementation of an IP video security system you should have regular updates and maintenance of your system. You should be demanding strong security – both physical security and IT security – of the video security system.

You did have IT involved in selection, implementation and operation of the video security system, didn’t you? You did make security a key part of the selection process, just as you would for any other IT system, didn’t you? You are doing regular security scans of the video security system and monitoring all network traffic, aren’t you? Good, you have nothing to worry about!

If you are like many companies, you are probably feeling a bit nervous right now…

For home and small business customers, a secure the camera approach simply won’t work.

  • Customer ease of use expectations largely prevent effective security.
  • Customer knowledge and expertise doesn’t support secure configuration or updates to the system.
  • The IoT vendor business model doesn’t support security: Low cost, short product life, a great feature set, ease of use, and access over the Internet all conspire against security.
  • There is a demonstrated lack of demand for security. People have shown, by their actions and purchasing decisions, the effective security is not a priority. At least until there is a security breach – and then they are looking for someone to blame. And often someone to sue…

Securing the cameras is a great recommendation but generally will not work in practice. Unfortunately. Still, it should be a requirement for any Industrial IoT deployment.

Solution 3: Isolation

If ignoring the problem doesn’t work and fixing the problem isn’t viable, what is left? Isolation. If the IP cameras can’t be safely placed on the Internet, then isolate them from the Internet.

Such isolation will both protect the cameras from the Internet and protect the Internet from the cameras.

The challenge is that networked cameras have to be on the network to work.

Even though the cameras are designed to be directly connected to the Internet, they don’t have to be directly connected to the Internet. The cameras can be placed on a separate isolated network.

In my next post, I will go into detail on how to achieve this isolation using an IoT Gateway between the cameras and all the other systems.

Posted in IoT, Security | Leave a comment

Representative IoT Device: IP Video Camera [Survive IoT Part 2]

One of the most flexible, powerful, and useful IoT sensors is a video camera. Video streams can be used directly. They can also be analyzed using modern software and an incredible range of information extracted from the images: motion detection for eventing and alerts, automobile license recognition for parking systems and theft detection, facial recognition, manufacturing quality control, part location and orientation for robotics, local environment for autonomous vehicles, crop analysis for health and pests, and new uses that haven’t been thought of yet!

The IoT revolution for video cameras is the IP (Internet Protocol) camera – a video camera with integrated computer that can talk directly to a network and provide video and still images in a format that can be directly manipulated by software. An IP camera is essentially a computer with an image sensor and a network interface. A surprisingly powerful computer which can do image processing, image analysis, image conversion, image compression, and send multiple real-time video streams over the Internet. The IP cameras use standard processors, operating systems, and toolkits for video processing and networking.

Modern IP security cameras have high resolution – 3MP-5MP – excellent image quality, the ability to see in complete darkness, and good mechanical construction that can withstand direct exposure to the elements for many years. Many of these IP Video Cameras have enough processing power to be able to do motion detection inside the camera – a rather advanced video analysis capability! They can be connected to the network over WiFi or Ethernet. A popular capability is PoE or Power over Ethernet, which allows a camera to use a single Ethernet cable for both network and power. For ease of use these IP cameras are designed to automatically connect to back-end servers in the cloud and then to display the video stream on smartphones.

These IP cameras are available with full support and regular updates from industrial suppliers at prices ranging from several hundred to a few thousand dollars per camera. They are commonly sold in systems that include cameras, installation, monitoring and recording systems and software, integration, and service and support. There are a few actual manufacturers of the cameras, and many OEMs place their own brand names on the cameras.

These same cameras are readily available to consumers for less than $100 through unofficial, unsupported, “grey market” channels.

IP cameras need an account for setup, configuration and management. They contain an embedded webserver with full control of the camera. Virtually all cameras have a root level account with username of admin and password of admin. Some of them even recommend that you change this default password… One major brand of IP cameras also has two hardcoded maintenance accounts with root access; you can’t change the password on these accounts. And you can discover the username and password with about 15 seconds of Internet research.

The business model that allows you to purchase a high quality IP camera for <$100 does not support lifetime updates of software. It also does not support high security – ease of use and avoiding support calls is the highest priority. Software updates can easily cause problems – and the easiest way to avoid problems caused by software updates is to avoid software updates. The result is a “fire and forget” model where the software in the IP camera is never updated after the camera is installed. This means that security vulnerabilities are never addressed.

Let’s summarize:

  • IP video cameras are powerful, versatile and flexible IoT sensors that can be used for many purposes.
  • High quality IP cameras are readily available at low cost.
  • IP video cameras are powerful general purpose computers.
  • The business model for IP video cameras results in cameras that are seldom updated and are typically not configured for good security.
  • IP video cameras are easy to compromise and take over.
    • Can be used to penetrate the rest of your network.
    • Can be used to attack the Internet.
  • There are 10’s of millions of IP video cameras installed.

So far we have outlined the problem. The next post will begin to explore how we can address the security issues – including obvious approaches that won’t work…

Posted in IoT, Security | Leave a comment

How I Survived the Internet of Things [Survive IoT Part 1]

Working with IoT from a software architecture perspective teaches you a lot, but leaves the nagging question “how does this really work?”. Theory is great and watching other people work is relaxing, but the time comes when I have to get my hands dirty. So I decided I had to actually implement an IoT project.

The first step was to define the goals for the project:

  • Hands-on experience with Industrial IoT technologies. I’m much more interested in Industrial IoT than Consumer IoT. I am not going to have anything to do with an Internet refrigerator!
  • Accomplish a real task with IoT:
    • Something useful and worthwhile; something that makes a difference.
    • Something usable – including by non-technical people!
    • Something robust and reliable. A system that can be expected to function for a decade or longer with essentially perfect reliability.
    • “Affordable” – a reasonably low cost entry cost, but with a bias toward functional capabilities, low maintenance, and long life. Balance initial costs with operational costs and minimize system elements that have monthly or yearly fees.
    • Secure – including system and network security. There will be much more on this topic!
  • Learn how things really work. Engage in hand to hand combat with sensors, devices, systems, wired vs. wireless, reliability, usability, interoperability, and the myriad other factors that crop up when you actually try to make something work.
  • A bias toward using commercial components and systems rather than building things out of Raspberry Pi and sensor modules. There isn’t anything wrong with Raspberry Pi and low level integration, I just wanted to work at a higher level.
  • And, to be completely honest, to have an excuse to play with some neat toys!

Based on these goals I chose to work on home automation with a focus on security and lighting. After considering many things that could be done I chose to implement monitoring of fire, carbon monoxide, power, temperature, water intrusion, perimeter intrusion, and video monitoring. I also implemented lighting control with the goals of power savings, convenience, and having lights on when you come home. When designing and implementing the various subsystems I chose commercial grade monitoring, sensors and controls.

I sometimes get the question “do you live in a bad neighborhood?” No, I live in a great neighborhood. The main reasons for this project were safety, reduced power consumption, and an excuse to play with neat toys. Yes, I got carried away…

October 2016: Things Attack the Internet

In October 2016 several large Internet sites were subjected to a massive DdoS (Distributed Denial of Service) attack carried out by hundreds of thousands, perhaps millions, of compromised IP video cameras and home routers. These attacks were some of the highest bandwidth attacks ever observed and are hard to defend against.

In January of 2017, an estimated 70% of the security cameras in Washington DC were compromised by malware and were not able to stream video. Workers had to physically go to each individual camera and do a fresh install of the original firmware to return them to operation.

Security experts have been warning about weaknesses in IoT for years. Many of these warnings are about how easy it is to compromise and subvert IoT systems. The October 2016 attacks showed that these IoT weaknesses can also be used to directly attack key parts of the Internet. A larger attack could potentially make the Internet unusable!

Since IP cameras were used in the first major attack by IoT on the Internet and I have several of these cameras installed in my system, let’s start our case study with with them.

The next article will begin exploring the capabilities, security, and business model of powerful and affordable IoT devices.

Posted in IoT, Security | Leave a comment

Red Hat Improves IoT Communications

The just released Red Hat Enterprise Linux 7.3 improves support for IoT by providing enhanced connectivity to IoT devices – better connection of “things” to the Intenet. To give a little context for this, IoT is based on machine to machine communication. IoT devices – sensors and actuators – talk to backend systems, usually in the cloud, using Internet protocols.

IoT devices are typically low-power, and use local communications. They connect to backend systems in the Cloud through an IoT Gateway. This IoT Gateway has to support both local communications to IoT devices as well as wide area networking connections to the backend systems.

RHEL 7.3 improves gateway to IoT device communications by adding support for Bluetooth LE (Low Energy), which is the most popular wireless interface used by IoT devices. RHEL 7.3 also adds kernel support for CANbus, enabling the use of high end scientific instruments and industrial controllers that use this interface. And, of course, support continues for the other widely used interfaces such as ethernet, WiFi, and RS-485 serial connections.

Connections to backend systems is typically done through ethernet, WiFi, or cellular networks. RHEL 7.3 upgrades to the latest version of Modem Manager, which provides enhanced support for cellular networks and cellular modems.

These improvements in communications provide a platform for building IoT systems. JBoss builds on top of this platform by providing middleware and message bus capabilities – a framework for building IoT applications and for using asynchronous enterprise message buses for connecting to backend systems. Red Hat is working on complete frameworks for IoT device connectivity, communication and management through the upstream Kura and Kapua projects in the Eclipse Foundation.

The result of all this work is a platform and open source ecosystem for building complex multi-tier IoT applications.

Posted in IoT | Leave a comment

IoT Gateways

After discussing the “thing” part of IoT in Devices – the “Thing” in IoT let’s take a look at overall IoT system design.

IoT Gateways connect IoT Devices to IoT back-end systems. Gateways connect to devices using interfaces like Ethernet, WiFi, Bluetooth, 6LoWPAN, RS-485 and CANbus. Gateways connect to back-end systems through the Internet, commonly using Ethernet, WiFi, or cellular connections. Gateways perform multiple tasks, including concatenation of multiple devices, protocol conversion, device management, and security. Gateways may also perform application processing.

Since IoT Gateways are connected directly to IoT Devices they have to be co-located with the Devices. This means that gateways are deployed in hostile environments. They are accessed through network interfaces connecting both to local devices and to the Internet. People have physical access to the gateways. Users need access to the gateway to perform a variety of functions such as device discovery and registration. These users may be inexperienced, malicious, or both.

Gateways will often need to function which disconnected from the Internet. Such disconnected operation may be deliberate – a low power sensor may only connect to the network once a day, and spend the rest of the time in a low power sleep state. A system on a moving vehicle such as a truck, train, or ship may have critical communications through an expensive, low bandwidth cellular link, and then intermittently connect to a high bandwidth link such as WiFi. This might occur when a truck pulls into a warehouse or service station, when a ship docks, or when a train enters a station. These systems would be designed for disconnected operation. Another case might be a hospital, which needs to continue operations, perhaps in a degraded mode, in events where network connectivity, power, and other resources fail. It is clearly unacceptable for a hospital to shut down if it loses connection to the cloud!

These situations mean that a complete software stack needs to be installed on the gateway, with all of the management, update, and access challenges that this presents.

While gateways will most commonly be structured as application specific appliances there are many ways to use gateways.

Posted in IoT, Uncategorized | Leave a comment