“Security is not convenient. Nobody likes change. But at the same time, we have to be secure. Finding that middle ground is the challenge in my position.” - Andrea Haddad
In this bonus episode, Bryson Bort sits down with Andrea Haddad, a leader in technology and infrastructure architecture. Andrea has almost two decades of global experience across infrastructure, including network operations, enterprise architecture, cybersecurity, and cloud strategy. After beginning her career on the IT side, Andrea now specializes in building secure, scalable, and resilient digital foundations. She takes Bryson through her transition from IT to OT, segmentation in the real world, and why the future of manufacturing depends on both better architectures—and better intentions.
What are the unique security challenges of a manufacturing environment? What are the best - and worst case scenarios for manufacturing and cyber conflict? And what’s the secret to overcoming cultural differences between IT and OT?
“In our industry, our main concern is protecting…. In people, protecting production, protecting society, environment. So we can work all day long to come up with new architecture, new ideas. But if there's another hacker that wants to hack into the organization, he's going to always find a way. So the magic wand, to be honest, would be just simply kindness,” Andrea said.
Join us for this and more on this episode of Hack the Plan[e]t.
This June 17-18, join us for Critical Effect DC! Register here: https://www.eventbrite.com/e/critical-effect-dc-2026-tickets-1987141703327?aff=oddtdtcreator
The views and opinions expressed in this podcast represent those of the speaker, and do not necessarily represent the views and opinions of their employers.
Hack the Plant is brought to you by ICS Village and the Institute for Security and Technology.
Andrea: So my name is Andrea. I am currently the Director of Infrastructure Architecture at a CDMO (Contract Development and Manufacturing Organization) company.
So I first started as a network engineer, a regular network engineer in an IT space, and every company taught me something to be where I am today. From an IT perspective, from, like, a telecom perspective, learning and growing on the network back in the days where firewall was purely just network responsibilities.
There was no, like, big cybersecurity department. Then growing into the opportunity of getting into a CDMO company, where it was a big shape of the perspective that I had for cyber, because when I learned what OT does, what critical infrastructure it has, it kind of led me to more and more understanding as to what that space is and how we protect it.
Because learning about it taught me that those are, like, critical infrastructure where people's lives are going to be involved and production failure, which is a very new space that I wanted to learn. So that's why I am where I am today, and this is how I got the opportunity to be with you today.
Bryson: For anyone who might not be familiar with our jargon, a CDMO is a Contract Development and Manufacturing Organization, a specialized partner in the pharmaceutical industry that provides comprehensive, end-to-end services, ranging from drug development and formulation to clinical trial supply and commercial manufacturing.
Bryson: Your background was IT and telecommunications, and then working at this first manufacturing opportunity was your exposure to the operational technology side?
Andrea: Yes, exactly.
Bryson: How did that start? So, I mean, you're there, and you're like: "Well, I kinda wonder what we do for a living," which hopefully was, you know, the start of it. I'm still amazed at how many folks work at some place, and they don't really understand what the company does, right? Like: "Oh, I know we're an electric utility."
Yeah, but do you actually understand, like, how the line workers, right, how the generation and transmission part works, that kind of thing? So what was your exposure, and how did that go?
Andrea: The fact that you work in IT, sometimes people ask me, like, "Which organization do you wanna apply for?" I mean, I don't care.
Everybody needs IT, right? So you just go and apply for a different organization, and then IT is IT. You do your thing. And then being in that space, seeing that, oh, my god, there's an unmanaged switch here. How did that happen? Oh, there's this user that's not in the domain. There's a general account here and there.
First of all, it was a shock to me, because to me, being in an industry where service providers are always IT focused, where all of those different industries that I worked with are very much into, like, having those standards set. Seeing that there was something different was the... something that triggered my curiosity.
And talking to the different team members in the OT space pulled my curiosity into, what are those? Then I start seeing, like, PLC name, and then I start seeing SCADA, and I had no idea what those were. So starting to, like, research to understand because they were always pushing back, like, we can't have a different type of switch.
We cannot connect to the corporate switch. Why do you have a cable connected to your desk? And then that desk has multiple other unmanaged switches that connect back to each other in a way that is very weird because a lab space is needed, and more testing is needed. So long story short, it was an educational journey to be where I am, and it was a lot of questioning, a lot, a lot of, uh, researching to be done to be able to understand what that space was.
And then obviously reading on critical infrastructure, reading on what pharmacy means, what CDMO means. I started being in different conferences that are not only pharmaceutical, but can be power. It's not only power. You know, like, it just opened up, like, the whole space of a different type of perspective that I used to have from an IT space.
Bryson: What have you learned about working at a CDMO and the operational technology side? What are the daily challenges that are unique?
Andrea: I'm still learning. I'm still learning. I'm not perfecting it yet, I should say. It's a continuous growth and a continuous surprise. It's just that the main, main, major concern that I had was vendor accessing from the outside, remote access.
When I was looking into the firewalls and seeing there are policies to allow vendor access from the outside, I'm like, "How are we allowing external vendors, suppliers to just connect to our network with a wide open rule?" It's like, no, we can't just do that. So it's not only, like, the computer, but it's also other stuff that would be involved.
So, I think this was the main concern that I had, the remote access. Obviously, there are a lot of other concerns, but I'm still struggling with the correct architecture for the remote access because the suppliers, to your point, the struggles, they don't want to abide by the rules. They wanna have their own remote access.
They don't wanna listen to what an IT person and such and such company is doing because they've been doing this forever, and they know better. So it's just like getting the ideas there, talking to the multiple suppliers, talking to the OT engineers, and to the VP of engineering. And I should say, I've made a lot of progress.
When I first came into the organization, I saw open rules in the firewalls for suppliers that are just any, any. And that's when I started the journey of finding a remote access solution, because to me this was very weird.
And the most weird of that would also be, the actual vendor, some of them would bring their own remote access solution, and then just connect from the internet. So this is a struggle that I feel is still happening, and we came up with a solution after talking to different vendors, that was not one hundred percent the way we wanted it to be installed.
But the struggle that we faced with the OT engineers was the inconvenience, again, of using it. Because you would log in from the outside using a special credential. Then, when you log in from the outside, you only have access to specific virtual machines that are in the OT space. And then from the OT VN's, you have your OT credential.
So this is already two credentials that the user has to worry about, different than the corporate credential. So this is just the scenario. You have three credentials that the user is taking care of. And yes, it is inconvenient. I don't wanna remember ten thousand passwords every day, so sometimes it ends up with them using the same password over and over again.
Also, making sure that we enforce the passwords, how strong they should be. So all of this comes together and adds more inconvenience on the users. So you ask about the conflicts, those are the things that the user doesn't like. They don't like to have the inconvenience there. So, walking through this is just convincing them that finding the right remote access would be the best solution for us.
So right now, we're doing it – the way we're doing it is a secure way, but I've also found an article on Waterfall where it's only remote access through unidirectional, which I would love to try because they are talking about it as being also like a hardware-enforced remote access solution. So this is going to be my next step into trying out if this will resolve some of the struggles, not only from the user perspective, but also from a security perspective.
So when I try to work with the engineers and with the organization, I like to make sure that I understand the user's struggle because I'm a user myself. I don't like to have ten thousand passwords to go in different directions, but at the same time, we have to be secure. So finding that middle ground is the challenge that I find in my position, because I always think like, after the problem is faced during the day, I think at night like, "What can we do better? How can we enhance? Is there any other idea that we can do?"
So that industry is not like once and for all, and you're done. It's like a nonstop, continuous research to try to find the right solution for the right purpose. Even if you have frameworks. Frameworks are just a theory on paper. You have to have a standard for every single component within that framework, and every single component within that framework takes a lot of time and effort and research.
I've learned a lot, and this is a two-way communication. So the idea of being able to learn from each other is important in both spaces. IT should learn OT, OT should learn IT, because IT has a different perspective than OT, and similarly, OT has a different perspective. So, understanding each other is the way to get there.
Bryson: And how do you do that?
Andrea: A lot of listening. I mean, I've been upset a couple of times, I should say, but it's more of being close to them and understanding their world. OT is... They're struggling every time something goes down. So if a production goes down, this is a lot of money that they are losing. When I was in the IT industry as a service provider, I would do an intervention on a Saturday at 8AM, and I don't ask anyone; I just send a notification.
That's it. When I got into manufacturing, I'm like, "No, I can't do this. It doesn't work that way." We have to wait for a whole plant shutdown to start, which is, like, every once in a year, or it's just, it has to be really, really thought of in advance with a lot of planning. But communicating to them...
So when I first joined a CDMO company, I had to, after my research was done, I had to present my analysis to the executive, to the engineers, kind of convincing them of what we have and what we can improve. And based on that, I kind of got the blessing and the approval, and I, I kept going. It's, it's a long journey.
It's not like a click of a button where everything is fixed. It's a journey that requires continuous communication and continuous learning. Even until today, there's a new thing that comes up every day that will require you to make sure you're up to speed, to make sure you know what's going on, so you can update even the architecture.
Even if the standard was set today, maybe tomorrow there's a new standard we need to think about because of something new that comes up.
Bryson: So you don't have to go into the details. We're going to protect the guilty. Because, of course, you were innocent in this. You mentioned you got upset a few times in those interactions with operational technology.
Can you give us an example of that? Because I think it highlights, while listening is a good place to start, there's going to be, with IT and OT having different priorities, you're going to find some conflict there. So I'd love to, as much as you can share, some of the details of that conflict and how it was resolved.
Andrea: OT legacy personnel are very protective of their system. They don't want anyone to touch it. They know better, which I agree, they do know better of their system. But the fact of making decisions without involving IT was one of the biggest cultural breaks that I was able to accomplish, where we have a couple of remote sites everywhere. So, and each one is separate.
So each OT engineer on that particular site would bring their own solution without involving IT, and then when the solution is actually implemented, it’s then we know about it. So those types of conflicts are when we actually want to bring that solution up to the network. How can you bring a solution that we never knew about, we never discussed the architecture of that solution? And we have assets that we lack visibility, we lack control, et cetera, et cetera.
So when I get upset, I sometimes take it personal, and I try not to, but it's just that, to your point, I don't wanna share a lot of details about what was happening.
Bryson: That's as much as I'll push you. Now that you've engaged with operational technology, and you've found your calling, of course, because doing something that ties to the technology that makes the difference, at the end of the day, OT is how the company makes money.
OT is mission. So what has been your involvement in helping bring security into that, and are there tools or frameworks that have helped you?
Andrea: It's, uh, so funny, Bryson, that you're asking me this question because a couple of years back, I was driving, and I was thinking, "What's my purpose in life?"
It's like, I work in IT. IT is so boring. Like, there's nothing good that I do. You know, like IT, okay, I connect people together, but really, what's my purpose? Like, I needed something that I am changing something in the world. I'm not gonna change the world, but I'm just doing a little thing that can change the world in a way. And then I bumped into the OT environment, and I started learning more about critical infrastructure, and that's when I realized that, yes, this would be my purpose, and this would be something that I would take it as a mission and, and just go, not only helping my own organization, but also helping the community in a way to bring awareness that this is something that we really need to make sure we secure it by design, which is to your question, what are the frameworks that we used?
And it's also... it's not that I knew it. I discovered it through learning the IEC 62443, the Purdue model. Knowing that I work in the network, the layering, the level zero three through, those are like the SCADA, like the industrial level, and then the DMZ level, the industrial DMZ level, then it goes up to corporate for level four.
So having that framework in place in the architecture made it easy to actually architect it based on specific region or a specific site and a specific equipment. I'm not gonna say it's one hundred percent abided by, however, anything new that is being built, I try my best to force this. And also, it wasn't only just from a design perspective on the architecture level.
I wanted to make sure we have standards at every level. So like what assets to use, what switches to use, what server, storage, et cetera. How should we connect to the cloud? How should we allow remote access? So following this framework and the standard is something that I'm trying to push towards in my organization.
Bryson: So let's talk about Secure by Design. This was an initiative popularized under the Cybersecurity Infrastructure Security Agency when Jen Easterly was the director. Do you feel that the initiative has succeeded? Again, this is a policy idea that's being done up here, and policy is always worth the paper it's printed on.
The reality is, does it help you at a company level? And is there work still to be done? If so, what is that?
Andrea: It definitely helps because it gives a perspective of how things should be, right? There are still a lot of questions because theory is different than practice, especially when there are... when there's like different scenarios, because you can't have a cookie-cutter everywhere.
You have to make sure that even though we follow the architecture, there are some places where we tweak it in a way. So going back to your question about connecting to the, like, what things can be maybe more clarified. I feel like the connection to the cloud is something that is still lacking. Especially now when I see all of those digital twin, and the new technology that are coming up from a future manufacturing perspective.
I start questioning, and I actually started a new project in my mind where, like, okay, if we, for instance, started this technology in our own organization, where should we put all of this, and how should we connect everything to the different clouds? So I have to say that they are helpful for sure, and I followed it, yes.
But there are some tweaking that have been happening. And also from a practice perspective, to your point, when we talk about leveling, sometimes we talk either logical or physical. A lot of times, if the company lacks financial resources, we kind of put the level three corporate and the level two connected to the same physical switch.
So I feel like it would be good if the framework were to specify physical versus logical in a way where things are more abided by, and just to make it clearer on every industry.
Bryson: Presuming that you are also very familiar with ISA/IEC 62443, why is that the standard that your CDMO follows, and are there any tips that you've learned for others in using that framework?
Andrea: So the reason why I kind of bumped into this is when I was talking to different integrator. I was talking to different integrator, and they would show me different topologies. The way I kind of got this was from different vendors as well. So when I was talking to these certain integrators, then I started looking for like the Rockwell designs, the Cisco designs.
And I started, like, kind of matching what everybody is doing. And then I found out that all of this was IEC 62443, and based on that, I kind of put the architecture standard. So there was no standard that the organization was following. It was more of a discovery of communication. Again, it was me not understanding. Because I was in a position where I needed to connect all of these equipment, I needed to assess the network, and I needed to enhance it.
And I was wondering why all of these OT engineers are connecting it the way they are doing. And through the communication and through the learning journey that I've done, which was also due to the OT engineers, because I understood what they were doing based on the integrator.
But the difference is that the integrator had the topology but had control of those switches. So the integrator wanted to install the topology, but wanted to own the switches, wanted to have all access control. The shift that I've done was, I agree, that design is amazing, but no, we need to take control as IT.
And following all of that, I feel like going to those different community conferences, kind of more and more affirmed that this is the way that we should be going, because it's good if you have a perspective from the inside, but it's also good to have the perspective from the outside. IT is sometimes isolated within the same organization, but also we should not isolate ourselves from different organizations.
Because if the community is all open, not every organization works alone, and we work all in a bigger community, we learn from each other.
Bryson: That is probably the most reasoned answer I've heard for 62443. Before we go into the lightning round, anything else that you want to cover?
Andrea: 62443, I didn't go into like segmentation, access control, et cetera.
Bryson: It's up to you. I mean... Actually, let's talk segmentation and access control.
Andrea: And also, like the weak authentications that we have, the domain structure.
Bryson: Yes. I love this part of the conversation because this also gets to the nexus of the IT/OT question, because those are typically purviews that IT is coming in, OT already has segmentation, access control done a certain way.
Back to Secure by Design, we have limitations on authentication that we're able to do. So could you talk through some of those challenges you've seen in, while 62443 is a great idea, the practical challenges of proper segmentation and access control are inherent conflict.
Andrea: Yes, I agree one hundred percent, Bryson. So there are some system that we discovered that don't accept domain, as simple as that. They need a generic account. And it gets difficult and, and it's like scratching your head, like, how should you fix that? What should you do? And sometimes there's no solution. Sometimes you just like, okay. Or that particular system does not accept any new patches or has a different Windows. And there's no money to replace it.
So in some cases, yes, it's very nice to follow the framework, but when you have some challenges and constraints like this, it's something that you try to fix but cannot fix. But at the same time, we've changed a lot of domains and a lot of access perspective because the company, especially when there is a big organization that acquires different organizations, IT has their own domain, to your point, and then OT have their own domain, and it's very hard to kind of manage all of this.
So the structures that we've done was to create an OT special domain managed by IT, and it also depends on the region. And this was a big shift because most of the way people accessed those devices was with a generic account. And the inconvenience of having security sometimes is also big, because people don't want to have three different credentials.
They don't like that. It's not convenient to be secure. And even though... even myself, sometimes, like, we have to log into multiple things to do something. It's not convenient. So security is not convenient, and changes, nobody likes changes. So long story short, access control is managed how much as we can manage it through the domain, but there are things, and there are assets that, to this point, don't have that capability.
Bryson: One of the common mistakes I see with IT and segmentation is not understanding the underlying processes, because going back to security is always gonna have a tension with convenience or functional use. I mean, because security can, and most likely will, introduce friction, and so properly architecting those segmentation and access controls can have as minimal friction as possible or can be so much friction that I always like to say, that's how you create the world's best hackers, is because when you make a control that gets in the way of an employee trying to do something, they're gonna just go around it.
Andrea: Yeah, I agree.
And then they use the same subnet over and over again. Every supplier has the same subnet, the 10-1-92 subnet, over and over again. And then, okay, how is that all working together? So it's a struggle when you are doing it on a day-by-day basis. It's a struggle because you want to be flexible enough not to upset anyone, but at the same time, you wanna try to convince them that, "Hey, I know it's inconvenient. I know that you're gonna have to log into such and such jump box to get into here, and I know that you're gonna have to do different credentials, but this would be better for the organization. This would make your system more secure," et cetera, et cetera.
So it's just like changing the mindset of the people, talking to the people, I see as the number one thing between the internal employees and the external suppliers.
Bryson: Okay. You ready for the lightning round now?
Andrea: Mm-hmm.
Bryson: If you could wave a magic, non-internet-connected wand, what is one thing you would change in the industry?
Andrea: You mentioned industry. Because I was thinking, to be honest, as simple as kindness. Because whatever we do from a security perspective, there's always an evil thing that will find a way to hack.
So in our industry, our main concern is protecting people, protecting production, protecting society, environment. So we can work all day long, come up with new architecture, new ideas, but if there's another hacker that wants to hack into the organization, he's gonna always find a way. So the magic wand, to be honest, would be just simply kindness.
Bryson: Can't we all just get along?
Andrea: Yes.
Bryson: You've waved your magic wand. Now, looking into your crystal ball, which looks suspiciously like an HMI, one good and one bad thing that you think is going to happen.
Andrea: I see how the future of manufacturing is happening right now, and I do really like it, because they're using all of these new technology, from AI to anything that is coming up. And they're using it in a good way where it's less waste, greener environment.
So the good thing that is coming up of all of this, at the end of the day, the technology is here to make people's lives better, societies better. And the way the future of manufacturing is going is exactly in the right direction. So I feel like the more this is going to grow, the more we'll see good things coming out of it for the environment.
Bryson: So what's the bad thing?
Andrea: Oh, yeah, the bad thing. Uh, the bad thing would be, honestly, and this is something I'm always afraid of, and it came up when I was doing my Harvard Cybersecurity certification, in my mind. It's like this is war. This is really like... We used to study history with, like, World War I, World War II, and now we're doing that digitally.
Like, if this is gonna continue, 'cause there are some articles that I used to read in that class where hackers are not only, like, hacking for financial purposes. They're just hacking to-- for a, a hospital to just take away somebody's life. Like, what's the point, right? So this is all is, like, in the next five years, the bad thing that I would see, this turning into a World War III that is digital.