“So our main product in Claroty is an idea solution. And in order for an idea solution to work properly, it needs to have a really good understanding and visibility into the protocols, to the network traffic. And so I started in Claroty as a protocol researcher, meaning I was trying to understand how industrial protocols operate, and this means I had to research a lot of ICS equipment to really understand what types of data, different components in the ICS network, exchange, how do they operate? What are the different protocols and how can we understand what they mean?” - Sharon Brizinov Sharon Brizinov is director of research at Claroty, a cybersecurity company focused on protecting industrial control system. In this episode, Bryson and Sharon cover Sharon’s career, his experience in the ICS industry, and more.
Claroty is a cybersecurity company that helps organizations to secure cyber-physical systems across industrial (OT), healthcare (IoMT), and enterprise (IoT) environments: the Extended Internet of Things (XIoT).
In this episode, Bryson Bort sits down with Claroty director of research and industrial control system (ICS) vulnerability expert Sharon Brizinov to discuss everything ICS.
What are the most common vulnerabilities threatening ICS security? What’s the impact of cybersecurity controls standardization? And if he could wave a magic wand, what is one thing he’d change in the ICS industry?
“Don't expose ICS equipment over the Internet,” Sharon said. “That's my wish. To eliminate all the ICS Internet-exposed devices.”
Join us for this and more on this episode of Hack the Plant.
Hack the Plant is brought to you by ICS Village and the Institute for Security and Technology.
I’m Bryson Bort and this is Hack the Plant.
Electricity. Finance. Transportation. Our water supply. We take these critical infrastructure systems for granted, but they’re all becoming increasingly dependent on computers to function. We walk through the world of hackers working on front lines of cybersecurity and public safety to protect the systems you rely upon every day.
From the ransomware threats of Colonial Pipeline to the failure of the Texas power grid, it is clear our interconnectivity is also a significant source of risk. This season, we will continue to bring you a panoply of different insights across all of the different things happening in critical infrastructure.
In my day job, I’m the CEO and founder of Scythe, and the co-founder with Tom Van Norman of the non-profit ICS Village, where we educate people on critical infrastructure security with hands-on examples, not just nerd stuff. I founded Grimm in 2013, a consultancy that works at the front lines of these problems every day for clients all over the world. I’m also an adjunct senior advisor at the Institute for Security and Technology, a 501(c)(3) think tank dedicated to tackling technology-driven emerging security threats.
This is Hack the Plant, brought to you by the Institute for Security and Technology and ICS Village. Subscribe wherever you find podcasts to get each episode when it drops.
I’m Bryson Bort, and this is Hack the Plant. For today’s episode, I’m joined by Sharon Brizinov, the Director of Research at Claroty, an industrial cybersecurity platform that protects systems across industrial and healthcare applications.
“So our main product in Claroty is an idea solution. And in order for an idea solution to work properly, it needs to have a really good understanding and visibility into the protocols, to the network traffic. And so I started in Claroty as a protocol researcher, meaning I was trying to understand how industrial protocols operate, and this means I had to research a lot of ICS equipment to really understand what types of data, different components in the ICS network, exchange, how do they operate? What are the different protocols and how can we understand what they mean?”
Sharon specializes in ICS vulnerability research. We discuss how he made the transition into hacking.
“…We’re talking about the early days of ICS security, everything was vulnerable. That's when we said, okay, there is a bigger picture here. It's not just about a minor issue. We needed to go over all the ICS market to each vendor one by one, find the potential bugs and vulnerabilities, and work hand in hand with the vendors to make sure the software, the firmware, the ecosystem is much more secure.”
And Sharon shares tips for other enthusiasts looking to get into industrial control system security.
First of all, if you have a hobby of hacking things, ICS is not any different from any IoT device that, normally people would start from. If you're a hobbyist and you're interested in ICS security, you can also buy a small PLC. So today you can go on eBay, and you can buy them and then you can start to understand the protocol. So you would set up a small PLC and you'll also download the engineering workstation and you'll start to communicate with the PLC to configure it.
What are the most common vulnerabilities threatening ICS security? What’s the impact of cybersecurity controls standardization? And if he could wave a magic wand, what is one thing he’d change in the ICS industry? Join us for this and more on this episode of Hack the Plant.
Personal shout out to Emerson Product Security - S4 for coming up and saying hi, steadfast audience!
Bryson: So who are you?
Sharon: Hi, so I'm happy to be here. My name is Sharon Brizinov from Israel, Tel Aviv. I'm a ICS hacker, I would say. I do a lot of cool stuff with ICS equipment from PLCs to HMIs to engineering workstations. Myself and my team, Team82 and Claroty. We work hard to make sure these devices are secured.
Bryson: And how did you get started in that? I mean, the average person doesn't wake up and find an industrial control system and just start going at it.
Sharon: Yeah, it's a good question. So I actually started Claroty just after the military service. So as you probably know, in Israel, the military service is mandatory for all men and women. So I was kind of in one of the cyber divisions in the military. So I had some cyber security background. And after the military service, I was looking for my first job after the military.
And my friend worked at an unknown company at the time, named Claroty, and he told me, ‘Come, it's really fun, we're drinking coffee and we have a PlayStation.’ And it sounded good to me. So, I applied and actually started in Claroty as a protocol researcher. So I'm not sure if all the audience are familiar with Claroty, but Claroty is a cyber security defense company trying to protect industrial systems.
So our main product in Claroty is an idea solution. And in order for an idea solution to work properly, it needs to have a really good understanding and visibility into the protocols, to the network traffic. And so I started in Claroty as a protocol researcher, meaning I was trying to understand how industrial protocols operate, and this means I had to research a lot of ICS equipment to really understand what types of data, different components in the ICS network, exchange, how do they operate? What are the different protocols and how can we understand what they mean? So for example, if you have a PLC, that someone is reading data out of it, for example, symbols or tag data, we need to understand how to read and understand the protocol.
And my job was to do that. To write kind of protocol detectors. And so with time, I gained a lot of experience and knowledge about ICS protocols. So that's how I started. And from there, since I had kind of a solid background in ICS protocols, I started to do some hacky things.
Bryson: So you got started working with protocols because Claroty needed to be able to interface and integrate with all of the many, many protocols that exist on an industrial control system network to be able to have that visibility and detection. And then one of the interesting things about a lot of those is the manufacturing specific ways that some of those protocols are implemented, right?
We have an RFC that says Modbus is this way, but then say, Rockwell decides to do it their own special way. How did you work through the fact that so much of this stuff really was trial and error and you had to observe it on the wire to figure out and reverse engineer the protocol itself?
Just like in real life, not everything has documentation; industrial control systems, whether they're the devices or the communication protocols themselves, are no different and manufacturers even create their own versions of protocols that deviate from the standard. So it's important sometimes to do reverse engineering, figuring out by trial and error and observation exactly how those things work so that we can interact with them.
Sharon: Yeah, actually, it's a really good question because I always joke that in ICS, you don't have edge cases. Everything is an edge case. So yeah, it's true, in ICS and OT networks, it's kind of each vendor did whatever they wanted. And so they kind of broke all the standards and each vendor created their own proprietary protocol.
Or if they had not created their own proprietary protocol, they abused one of the standardized protocols and used it however they wanted. For example, some vendors took Modbus and did some kind of their own flavor of Modbus. For example, Schneider did their own flavor above Modbus and some other product vendors created their own proprietary protocol for their own specific use.
So, yeah, that was the challenge actually in the early days of when I started to do protocol research was to understand the variety of these protocols, which some of them look very similar, but they're different. And we needed to support all the edge cases for all the different types of equipment.
And it was very difficult and hard work that we had to do. We had to acquire most of this equipment, plug it in in our lab, understand how to properly configure it and simulate how users would use it and start to understand how it works. So we had to do this for every major platform, and, I don't think you have been in our lab yet, but you're more than welcome, and I'm happy to show you the hundreds of PLCs and HMIs and the different equipment that we have. This is exactly how we were able to support all of the different flavors and protocols and proprietary mechanisms. We just went one by one and we did a lot of hard work.
Bryson: Yeah, I would look forward to that. The last time I was in Tel Aviv, I was there less than 77 hours on the ground. And it was during COVID, so I was quarantined for part of it as well. So my next trip, I hope I get to stay there a bit longer and quarantine is not as onerous. Okay. So reverse engineering, a lot of these undocumented protocols for product development tripped you into then deciding to do in quote ‘hacky things.’
So tell us about that story. When did you first decide to go above and beyond and what was the motive?
Sharon: Yeah, sure. So. I started as a protocol researcher. So with time, I learned how different ICS protocols work and to really understand how an ICS protocol works, you need to do two things. So first of all, you need to do what we call blind protocol analysis, which is basically have a setup in your lab and you, you use kind of a sniffing tool to capture all the network traffic and you go over the, the protocol, the network traffic protocol, and try to make sense out of the bytes. Because eventually it's like a language. You have very structured bytes and you need to make sense out of it. So the blind protocol analysis approach is where you look at the bytes and you start to understand, okay, this is the header.
This is the function code. These are the attributes and you start to break it down packet by packet until you have, like, a basic understanding of the protocol, its structure, its layout, et cetera. So that's kind of the first approach. And the second approach is to do a more in depth analysis and more in depth analysis means in most cases, if it's a proprietary protocol and you don't have a document or RFCs, is to understand how the software and the firmware works.
So you kind of look at the software from the inside, look at the firmware, try to understand where the packet handling functions are located, you do in depth research into these functions, try to understand the assembly of how packets are being handled. And then you kind of follow the disassembly trails to understand the handling of these packets. So for example, if a user clicks on the engineering station on the read program or kind of, reads the tag or upload or whatever functionality that the engineering station allows them to do, then eventually multiple packets that says please read this tag or program or do the upload procedure will go on the network line and these packets–whenever they're being received by the other side, in this example, the PLC firmware–they will be ‘handled,’ which means the PLC will read this packet and will understand that it needs to do something. And if you look at the disassembly of these functions, you can understand what the correlated bytes mean. So whenever we started to look at these areas in the code, we started to notice some funky things, for example, uninitialized pointers or potential… potentially buffer overflows.
So at the early days, you know, again, we're still doing protocol research, to gain visibility into the ICS networks. So we thought to ourselves, okay, what happens if instead of this byte, I will send another byte, like a bigger one it probably will trigger a buffer overflow, right? So we tried it and it worked, the PLC crashed and we thought to ourselves, yeah, it's very, very interesting behavior, why there are no protections against it. And so we reported it to the vendor, vendor said, ‘oh, wow, I didn't know about this, thank you’ and we moved on in our life. And then again, we came to the lab in the following week and we found a similar behavior in another vendor. And so we triggered the bug. We caused the PLC to crash.
And I think once we got to the 40 or 50, buffer overflow. Again, we're talking about the early days of ICS security, everything was vulnerable. That's when we said, okay, there is a bigger picture here. It's not just about a minor issue. We needed to go over all the ICS market to each vendor one by one, find the potential bugs and vulnerabilities and work hand in hand with the vendors to make sure the software, the firmware, the ecosystem is much more secure.
So that's how I did the transition from a protocol researcher to an ICS hacker.
Bryson: Okay. So reverse engineering protocols, because they weren't documented to disassembly to then deciding to do packet injection for buffer overflows. And you made the comment back, this is the early days, which by the way, the early days are not that long ago for industrial control systems security and everything was vulnerable.
And the implication being everything is not still vulnerable.
Sharon: Well, I can, I can assure you that now I think, we recently recorded our 550 CVE. So now everything is much more secured. At least the low hanging fruits are over. Because we did a very thorough job on going over all of the major vendors, all the kind of, the popular equipment, and we reported all the bugs we could find.
So, yes, I think that today ICS equipment, even the old equipment with updated versions, are much more secured, not to talk about the new generation ICS equipment, which is secured by default in most cases, most certainly for the major top vendors like Rockwell, Schneider, Siemens, et cetera. So, so yeah, today the ICS equipment is built with the security in mind.
And I'm sure that the updated equipment is much more secure.
Bryson: Okay. So starting from finding a lot of low hanging fruit, which buffer overflows are pretty basic, to over 550 CVEs found, you believe that, that plus other researchers have now pushed the manufacturers to better secure coding principles, and so the kinds of vulnerabilities you're finding now are much more difficult than your experience.
Sharon: True, true, true, true. Yeah. So if in the past it was, we were able to trigger a buffer overflow vulnerability by accident, you know, just by fuzzing different packets on the network, today, it's much more difficult. We're using much more resources, time, and money to analyze modern firmwares, it's much more difficult because these products are built with the security in mind and we need to invest a lot of time to understand kind of, to be creative and find how to circumvent modern security enforcement in order to find and trigger more bugs. So, I'm pretty sure not just Claroty, but all the ecosystem, including other research groups, we have come through a long, a long path. And today it's much more difficult to find bugs in modern ICS equipment.
Bryson: What is the most common vulnerability that you find now?
Sharon: So, yeah, in the past, it was 1 percent buffer overflow. So for example, think about this way, you want to read a tag, ICS tag from one of the PLCs. So you ask the PLC, ‘hey, can you give me this tag?’ And it will give you the tag. And then you ask, okay, can you give me an array of tags for, this is just an example, and then it sends you an array of tags and then you ask, okay, can you give me an array of nine, nine, nine, nine, nine, nine, nine, nine, nine, nine, nine tags. And then it crashes because of a buffer overflow because it tries to allocate too much data and it overrides part of the stack or heap in some cases. So buffer overflows were very common in again, the early days of ICS security. Today, things are more complicated.
So I don't think there is a common vulnerability that I can pull out of my mind right now, but we are seeing some cases of security features that vendors are trying to enforce. And we are seeing some cases that we're able to circumvent these security features or bypass them because the vendors did not think about some specific edge cases, like what happens if you take this PLC and you plug it in in a very specific way and you configure it in a very specific way. So I think today the more common bugs are to bypass security features.
Bryson: You mentioned secure by default. What other security features have you seen come out from the manufacturers?
Sharon: Yeah, good question. So I think the best security features we've seen from the major vendors are TLS usage. So in the past ICS protocols were kind of, no encryption and no, no authentication, just like plain bytes sent over the line. And thankfully, because of the research community and the change and kind of the shift in the industry, the vendors now are starting and already started to implement TLS on top of their protocols.
So now, for example, if you want to communicate with the Siemens equipment, everything runs by TLS. And TLS is the industry standard, not just for OT now, but also for IT of course. Everything runs over TLS, which means the traffic is much more secure and you cannot even communicate with the PLCs without having the proper certificate or at least a shared password.
So for sure, the most important security feature that was developed in ICS in recent years is the usage of TLS. And you can see TLS in Siemens, for example, since TF 17, everything is secured over TLS. Rockwell are using SIP secure, which also runs over TLS. General Electric as well are using TLS. This is definitely not only the future, but also the present. And I'm very happy to see this transition.
Bryson: So I'm going to throw you a curveball here because it's coming from the other side. IEC ISA 62443 has become the adopted golden standard for product security. How do you see how that framework has impacted this space?
Sharon: So I'm not sure if you're talking about the programming side of it, like how to write secure programs.
Bryson: Well, so it's pretty comprehensive. It doesn't necessarily say this is how you write secure programs, but it creates a framework. just like ISO 9001 creates a framework for quality outputs from a process control for manufacturing. 62443 is looking and going, ‘what is the framework for instilling cybersecurity and best practices for product security, and then the operation of the products?’
Sharon: So I think overall, the fact that we are talking about a standardized, a standard in this field of security is always positive because the vendors, when they're now thinking about the new generation, PLCs, new generation, HMI, new generation equipment, security is not a nice-to-have feature anymore.
Security is part of the development life cycle of every product, and they understand they need to stand in some kind of, minimum security standard in order to have a product in the market. So these standards and the security development life cycle of this product makes the entire ecosystem and the entire development of these products much more secure.
And again, at the end, it's not only the benefit of the asset owners. It's also the benefit of the vendors because they know they can trust the equipment and they do not need to release emergency patches every other day. So I think overall, everyone benefits from a secure product, and these standards push the security of these products much higher.
Bryson: Folks are always trying to understand, how can we do this at home? So this is going to be a two part question. The first is for the amateur enthusiast, the practitioner who wants to start to experiment, what are the recommendations you would give for how to create a home lab? And then you mentioned the extent of the Claroty Lab. What are some recommendations you would give to an enterprise for building that level of a lab?
Sharon: Yeah, okay, so, for hobbyists, first of all, if you have a hobby of hacking things, ICS is not any different from any IOoT device that, normally people would start from. So, usually people are buying like old camerad, IP cameras or old IP routers and started to do some hardware hacking, you know, using UART or other type of equipment to extract the flash and read the chip and start to do some firmware analysis. So very similarly, if you're a hobbyist and you're interested in ICS security, you can also buy a small PLC. So today you can go on eBay, and look for Rockwell PLCs or Schneider PLCs. Usually they're not very expensive. There are some expensive PLCs, but we're not talking about them today. And you can buy them and then you can start to understand the protocol. So you would set up a small PLC and you'll also download the engineering workstation and you'll start to communicate with the PLC to configure it.
Obviously there is a learning curve to understand the terminology, the different softwares, how to configure it properly.
UART is a universal asynchronous receiver transmitter. It defines a protocol for exchanging serial data between two devices. UART is very simple. Just has two wires between a transmitter and receiver to transmit and receive in both directions.
Once you're over that, you can start looking at the protocol and see what happens if you send a different kind of a byte sequence and start to see how the PLC behaves. So this is exactly how we started, right?
We bought equipment. We plug it in our lab and we started to play with it to understand the ICS protocol and then to poke it around and send different sequences, different bytes and see how it behaves. So I think after that, going over the firmware would be the next step. Usually in ICS, the firmwares, are a little bit tougher to work with because they're not, in most cases, they're not based on like a Linux kernel.
In many cases, they are more RTOS based, like VxWorks or QUnix Or other flavors of real time operating systems.
RTOS, a real time operating system for real time computing applications that process data and events that have critically defined time constraints. A RTOS is small, fast, responsive, and deterministic, which means it executes tasks quickly, efficiently, responding at an expected time.
But if you're really into ICS hacking and you want to explore RTOS firmwares, there are some examples online and you can always, ping us, Team82 our research team, and we'll be happy to help you, start with your research.
And regarding the second question, the enterprise. So it depends what type of enterprise are you. So if you're a security vendor enterprise, focused, mostly in IT, but you want also to gain some understanding and visibility in OT, then yeah, I definitely recommend, to start buying some ICS equipment, even the more expensive ones and do exactly what I mentioned earlier with the hobbyist. If you're a vendor, ICS vendor, then obviously you probably do have already a lab you just need the right people that have motivation to do some hacking and then, you'll start to gain more visibility into your equipment and gain more understanding of what are the potential security flaws your products might have, which obviously will help you to fix it and secure your product.
So overall, I think buying equipment on eBay. And start to configure it and play around with it. I think anyway, it's a good start for anyone who is interested in security, not only ICS security.
Bryson: I will add one piece of wisdom and I'd like you to confirm, which is anytime you're doing hardware hacking or particular industrial control systems, be prepared that you're going to brick it.
Sharon: Yeah, definitely. I've bricked a lot of ICS equipment. That's for sure.
Bryson: So have more than one on hand, cause you'll break it. And by break it, I don't mean you blue screen it, you turn it into a paperweight.
Sharon: Yes, this is correct. There are, I mean, at least in the past, it was very, very easy to brick ICS equipment because again, there were no defenses, the vendor expected users to use it as it recommends and anything else might brick it. So hopefully it has changed a bit and the newer versions of the firmwares are more protected.
Bryson: A funny story that relates back to your origin story with the Sony PlayStation, as the Sony PlayStation was the first PCB I ever bricked.
Sharon: Oh, wow. Expensive one.
Bryson: It was an expensive, yes, at that time, on my budget, that was a big deal. I was very upset. Soldering is not my strength.
Sharon: I bricked a lot of PLCs, a couple of HMIs. I mean not the entire PLC, mostly specific cards, because in PLC, you have like a rack and different slots and different modules. So I bricked like specific modules but yeah, I also bricked N HMI like two weeks ago, because whenever doing vulnerability research, so if you find a vulnerability and you exploit the device, and then you play around with the internal operating system, if you're not careful enough, you might brick it because you change how the device boots, and then you have no other way to recover it other than rewrite or reflash the chip. So until then it's bricked.
Bryson: Do you have any kind of ceremony or a little device graveyard that you do when this happens?
Sharon: Oh, we do have a graveyard. Yeah, we have a box.
Bryson: You do keep a graveyard?
Sharon: Yeah, we have a box with all the dead devices.
Bryson: Okay not quite ceremonial, but I guess still there for, for memories.
Sharon: Luckily for us, in most cases, it's quite easy to buy equipment on eBay. So, it's not a big deal in most cases. Yeah, I'm not, I'm not talking about $10 K U.S. dollars equipment. Yeah. But by the way, interesting point, whenever we stumble upon a very expensive equipment, like I mentioned, recently one of our team members, Vera, she researched a 100,000 US dollars worth of equipment, it's a gas chromograph fed by Emerson.
And since we did not want to work on production, because we did not want to break a $100K equipment, she actually simulated everything. So she emulated the firmware and she did the vulnerability research on the emulated environment. and she actually created like a software-based emulation of the entire equipment, including some of the peripheral bus, and the operating system, and the firmware, and everything. So, this is another great way to start the research from emulation. But it requires a lot of experience, to be able to do it correctly. But yeah, emulation is a great way to start as well.
Bryson: All right going into the final two questions. This is where you got to put your big thinking hat on. Are you ready? Larger strategic implications. How we end every episode. If you could wave a magic non Internet-connected wand. What is one thing you would change in this industry?
Sharon: It's a good question. So in the past, if you would ask me this, like, five or six years ago, I would say vendors, please work with security researchers. That's definitely my wish five or six years ago, but things have changed since, and now all of the major vendors are much more appreciative whenever we report them a vulnerability.
So I would say that probably I hope that asset owners will not connect ICS devices over the Internet. Don't expose ICS equipment over the Internet. That's my wish. To eliminate all the ICS Internet-exposed devices.
Bryson: Okay, Censys and Shodan will be very disappointed.
Sharon: For sure. But, but yeah, we'll see less hacking going on.
Bryson: All right, Sharon, you've waved your magic wand. Now, looking into the crystal ball for a five year prediction. One good thing and one bad thing that you will think will happen in about five years.
Sharon: Okay. Good thing is we will see TLS more and more. So all the major vendors will enforce TLS over their protocols. So ICS networks will be much more secure and we will see much more of a TLS traffic. The bad thing is, we'll see much more ICS equipment connected to the cloud. And this is the new modern ICS equipment is in many cases, cloud based.
So you can see all already, for example, Codices, you can configure and remotely control Codices PLCs over the cloud. So you can see it anywhere in the world, the PLC is connected to the cloud and you can control and monitor it and even do upload and download to different programs over the cloud. So unfortunately for ICS, I think it's a bad thing because it opens a whole new set of potential harm to ICS networks.
Bryson: This is Hack the Plant, a podcast from the ICS Village. Catch us at an event near you. Subscribe wherever you find podcasts to get episodes as soon as they’re released. Thanks for listening.