I had the pleasure of working with the team at Clutch Events to organise a Sydney event for experienced application security professionals. I opened the day with a keynote discussing core personalities encountered during an AppSec career and how to best manage their worst tendencies. Throughout the day, I spoke with delegates from technology firms, banks, superannuation, professional services, health, government, and plenty of other verticals.
In this post, I wanted to answer the most common questions that came up and share my knowledge with the wider community.
Question 1: How did you learn public speaking? Is there something I can do to get better?
This came up frequently. Most people feel terrified at the concept of putting themselves on a stage and setting themselves up for judgment from their peers. I think the best steps people can take to get better at public speaking include:
- Know your content. I’ve worked in the AppSec space for a very long time. When you know your content, talking about it becomes second nature. Where I see people fail is talking from speaking notes or off the slide deck itself, memorising a speech and faltering when they lose their place, or otherwise getting caught out by disruptions like an audience question. The best way to prepare for all these scenarios is simply to learn your material.
- Start. Like getting fit, the hardest part is getting off the couch and into the gym. You don’t need to immediately try out for TED talks or BSides Canberra; you can try talking to your peers inside your workplace, meetup groups, or panel discussions until you gather experience.
- Record and review. If you’ve got a friend who can take a video of your presentation, or the conference does, that’s amazing. If we watch our own videos, we can make huge improvements by identifying habits or issues with our presentations. You should never analyse your presentation while delivering it, you’ll just perform worse. Watch the recording a week or two later, and take note.
Some common immediate issues I see:
- Stuttering, Ums, and Ahhs. Generally, you will speak with conviction if you have fewer conjugate sounds. That said, it’s not an easy habit to break, especially when normal conversation is full of these. Slowing down is the absolute best way to eradicate this.
- Fidgeting. Just don’t bring your phone, ID card, wallet, lanyard, etc., onto the stage.
- Going too fast. Drink water, pause at the end of sentences. Almost everyone goes too fast, it gets better with time.
- Avoiding the audience. Hiding behind a podium, staring at the sky, wandering aimlessly, looking at slides. All of these create barriers between you and the people you are speaking to. Try to recognise when you put an obstruction up.
Lastly, remember that everyone respects you for being an awesome human and giving your knowledge back to the community. They all understand it’s terrifying to even stand on a stage and speak. This always helps me get prepared to give a presentation. I hope this helps.
This video was also useful for me!
Question 2: Why are people so interested in the Software Bill of Materials (SBOM)? I don’t understand the relevance. Can I just give SCA results?
The SBOM is a nested inventory of items that make up software components. Having an SBOM helps you understand what risks you inherit in your supply chain and allows you to respond faster to a new vulnerability.
It is often a mandatory regulatory obligation too. The CISA proposed the concept in 2018, and the Executive Order on Improving the Nation’s Cybersecurity, dated May 12, 2021, enshrines it in US law.
Generally, it gets a lot of focus because of that EO. In Australia, we don’t have the same regulatory obligations, but many of the software companies that export to our nation try to adhere to their largest markets, which include the US.
I anticipate that the ISM, APRA CPS Guidelines, and other regulatory bodies will slowly mandate SBOMs, so it doesn’t hurt to look into them. I’d get started by:
- Establishing a software asset inventory. If you use SaaS providers, ask them for an SBOM. If you build software yourself, identify which matters for your business and who is responsible for their maintenance.
- Using automation to generate an SBOM. While you could just take a package-lock.json or similar, this only helps identify the top level packages and not transitive ones.
- De-duplicating/aggregating results. This can be a challenge. It’s easier to store independent SBOMs for each core software application.
On the note of SCA, they generally map vulnerabilities to software libraries in an SBOM. If you’re asked to produce an SBOM, you probably shouldn’t blindly give SCA results and expose your vulnerability posture more than necessary.
Question 3: Security is a bottleneck for my organisation. Why is this the case, and what can I do about it?
It depends on the organisation’s operating model. You can’t build an effective security program that functions independently of the business’s operating model. At a high level, I like to group things into product delivery and project delivery models.
Bottlenecks occur mostly in project delivery models. That isn’t to say that security doesn’t also create productivity impacts on product delivery models, they’re just less visible.
Assurance activities tend to be blamed for most project delivery bottlenecks. Penetration tests, code reviews, DevSecOps tooling, etc., tend to identify issues as a project reaches closure, and the bottlenecks arise from having to rework or remediate findings. This isn’t the fault of the testers generally; it’s the project delivery team’s fault for making decisions around quality and time that have security consequences down the track.
We frequently talk about shifting left, and it’s still fairly true here. Using technology or architectural patterns with inherent security properties, training people to deliver higher-quality software, and using tools to identify and plan for addressing software defects all lead to higher-quality software, which will have fewer security issues. It’s about enforcing ways of working for project delivery teams that make these priorities.
For product teams, it’s less visible but generally boils down to context switching killing productivity. When developers need to manage operational activities like patching dependencies while simultaneously working on feature delivery, both are performed slowly. The bottleneck may not be visible on a sprint by sprint basis, but the velocity and cadence of delivery over an OKR will be noticeably lower than other teams.
If you’re using a product delivery model, consider bundling operational activities into a time-box block and focusing on feature delivery. Your developers and product owners will thank you.
Question 4: Security Champion programs seem to start with initial excitement but drop off pretty quickly. Why is that the case, and how do I retain champions?
The simplest answer is incentives. Most Security Champion programs are created without considering what benefit there is for champions to participate and instead rely on goodwill, positive intent, and people’s keenness to learn. This helps identify an initial cohort, but ultimately, if they’re responsible for feature delivery and not receiving some tangible benefit, they’re likely to drop off.
The incentives I’ve seen that matter are formal training, delegated authority, expedited security processes, rotations into security, access to security leadership, and plain old cash. People who are rewarded with the training they’re interested in, such as a CSSLP or a pen test qualification, will want to continue participating in the program and be better champions (or recruits) for it.
Delegated authority and expedited processes can be problematic. Generally, there is a separation of duties between the three lines of risk, and giving engineers the ability to mark their own homework can be problematic. Smaller businesses won’t be concerned as formalised risk reporting isn’t present, and technology companies will consider velocity and autonomy where possible.
Access and rotations are interesting approaches. Secondments and formal mentoring can be great ways to have consistent buy-in over time and to consider bringing the right people into your recruitment pipeline. Just be aware that their incumbent management team might be wary of poaching.
Cash incentives work the least, simply because they won’t attract the right people to apply to be security champions. Everyone wants money, but you want people who care about security.
Ultimately, I’d start thinking about the incentives first if you’re worried about retention.
Question 5: I’m a developer, and security teams seem to have no trust whatsoever for me and my peers. Why is that the case?
We’re in the business of helping people identify problems and make informed choices about those. I often see security professionals, especially as they gain more experience, become cynical and jaded about the quality of the software they encounter. This isn’t healthy, but the industry doesn’t help people get out of that echo chamber.
Malicious compliance, intentional corner cutting, and incompetence are cases where distrust is warranted. Most developers are trying to do the right thing, and it is exceptionally rare that someone writes software with the intention of making it bad.
Malicious compliance is interesting. There are many examples I’ve seen in my career. For fun, here are a few:
- Including a -Exclude **/* as the job name so that the DevOps pipeline that runs the static analysis job appends that string and avoids scanning all folders and files. This is particularly fun because the AppSec team aren’t doing input sanitisation.
- Providing code that isn’t representative of production, often removing integrations or features that only operate in production.
- Renaming or overloading variables and functions to avoid syntax-checking rules. For example, EMDEEFIVE, Passwourld, StringComply, quay.
- Commenting out functional code to prevent AST creation, especially for checking taint propagation through code.
If individuals do this, they suck. But also, it’s a good indicator that the process you have created is ineffective and needs to change. It’s worth discussing with those engineering teams why they try to work around the controls and create a happy middle-ground between delivery velocity and security.
Security cannot solve corner-cutting and incompetence. Ultimately, the people driving projects or products make decisions to lower the software’s quality in favour of cost and time. To change this, you would need to have processes where security can improve delivery cadence or reduce costs, such as platform engineering approaches.
That said, the best way to establish trust is to spend time with the security function, learn from them about their concerns, and try to help. Dig the well before you’re thirsty.
Question 6: It seems most of the providers here lack a domestic presence for support with their products. Why is that the case?
It comes down to company valuations. Most of the software security companies in the APJ region started in Israel, San Fran, or London and took capital to expand globally. There is an expectation that when a business takes capital, it needs to make a return on investment that is significant for the investors. Expanding to new markets is one of the easier ways to do this.
The other major factor is company valuations. Software companies want to be valued based on EBITDA and ARR, as it allows them to register valuations of 8-15x those numbers. When you start employing people, it significantly reduces the margins you can operate with. When you start collecting services revenue, that part of your business will register a valuation of 2-3x. In general, any sensible investor will want to have close to zero services revenue to maximise their valuation multiplier.
Because of this, most software companies would prefer to find service delivery partners and work with them rather than employ internal professional services. In the AppSec space, there are very few domestic partners with this capability because the space is still nascent, and security is otherwise dominated by penetration testing, managed SOC, and GRC services.
Question 7: Legacy. What do I even do about securing legacy systems?
It comes up a lot. Generally, it’s a business decision and not within your control. You’ll encounter plenty of zOS, CICS, and RACF on mainframes. You’ll also see Perl scripts deploying onto physical servers, classic ASP pages serving Silverlight for LMS, and PHP 5 timesheet systems. The main thing is to recognise that while these things are horrendously insecure, they aren’t the priority. Inform, and give options, but don’t become emotionally attached to the idea of removing legacy. After all, the code we write today is our legacy tomorrow.
If you do want business buy-in, though, then you need to articulate in terms of cost, resilience, risk, and time. If the system is simply too expensive to keep operating, there are processes where a business continuity plan simply isn’t enough, where criminal liability or a threat to human life is present, or where there is a clear cut-off, such as reaching EOL, these are great places to start with helping build a business case.
