Product Security in Agile Organizations: part 1 - Empowering Teams

Product Security in Agile Organizations: part 1 - Empowering Teams

So you’re part of the security team in an Agile organization? The development teams ship new or updated code and infrastructure at least once every month, and up to several hundred times a day. Automation can solve or highlight a large amount of potential security considerations, but maybe an even larger amount of potential issues would not get caught. As security, your job is to figure out how to ship secure products whilst not negatively impacting your organization’s ability to remain agile and quick on their feet. How does your security team work with the development teams?

For some security teams it feels like this: the tools and processes you offer are used by only some of the development teams, but those who don’t want to partake in your security “burden” will happily work around you and ship their product to their customers anyways. While in this case the development teams remain highly agile and able to move fast, they can deliver potentially insecure products, making you and customers sad pandas.

For organizations with more strict security processes and workflows to follow, the security team can easily become a bottleneck if a large amount of that service work needs to be done for your development teams. While you’re now shipping secure products, your development teams are losing their agility in often highly competitive environments.

Both of these are extreme ends on a spectrum, but both are undesired and unfortunately happen a lot. Trying to find a nice balance on this spectrum is not very scalable, as one way or another, you’ll either slow down and reduce the agility of your organization, or you’re shipping potentially insecure products. What can we do?

Re: Agile

Let’s go back and take a look at what Agile was meant to solve. Waterfall methodologies required frequent handovers (aka “throw-over-the-fence”) between various functional departments (e.g. Development and QA and Design). In Agile we create small teams of all the various skillsets, allowing them to ship a product or feature completely on their own. By doing this in small iterative increments a fast moving, highly agile organization emerges.

Going into a little bit more detail:
In order for an agile development team to be able to have a potentially shippable product, they write code for the frontend, backend, UX/UI design, QA testing, release, etcetera. Every single one of those activities that the development team is able to do themselves is what they typically include in the Definition of Done for their items. Agile organizations will always be pushing for having the Definition of Done to be equal to the potentially shippable product. Why? Because as soon as those two differ, development teams are no longer able to ship products on their own as they depend on other teams in order to ship their items, and the organization loses its ability to move fast and remain agile.

An example: a development team is building an application. Their Definition of Done includes items such as backend and frontend development, UI/UX design and QA testing. When that’s done they’ll hand it over to the release team. In this scenario, the development team is being blocked by the release team. The organization would usually make efforts to integrate release engineering knowledge and capabilities into the development team, so that they do not have a dependency on the release team. So, we never want to make security requirements part of the activities for a potentially shippable product, without giving the development team the skills and capabilities to act on those activities.

Back to Security

As the security team, there’s a significant amount of requirements, activities and tooling you want the development teams to go through before they release their software. Think of activities such as pentesting, threat modelling, architecture review, ensuring TLS is properly configured, infrastructure is correctly configured, etcetera. If we introduce these activities as a requirement for a potentially shippable product, but the development team does not have the knowledge to successfully conduct or implement your security activities and requirements, they either will simply work around you and potentially ship an insecure feature. If they depend on your security team to help them do these activities, this makes your security team a blocker, and now you are reducing the ability of your organization to move fast and remain agile. If a team’s Definition of Done includes items such as “Requested X from Security” or “Security has done Y”, this is most likely the case.

This missing piece that allows development teams to remain agile, increase learning and improve security is to push security knowledge, perspectives and capabilities into the teams. On big and high-risk projects and products you can do so by embedding a security subject-matter experts into the team, on all other development teams you want to help them develop the skills and knowledge they need to build safe and secure products. This requires more relationship building and practice of “Go See” (aka Gemba) with development teams while also providing them with specific and relevant training materials.

As the teams are learning and getting better at security, they’ll slowly but certainly progress towards passing the security bar that you established. As the teams are improving, slowly chip away at mandated processes and workflows to increase agility but keep the overall product security level where it’s at. As your overall security posture should now improve, you may want to think of doing red team exercises and bug bounty programs.

Hopefully, the end result

Now, your organization is doing what you want it to: deliver safe & secure experiences to your users whilst remaining agile and fast-moving.

PS: This is my first blogpost and accompanying YouTube video. I would appreciate feedback on both the content as well as style and overall quality.

About Koen Hendrix