The cloud-native, all-remote security challenge

Heather Simpson ·
Sep 13, 2019 · 9 min read

We sat down with GitLab security engineer Jayson Salazar to talk about the challenges of working in security ops in a cloud-native, all-remote company like GitLab and the security myth he thinks should be debunked.

Name: Jayson Salazar

Title: Security engineer, Security Operations

How long have you been at GitLab? I joined GitLab in January 2019

GitLab handle: @jdsalaro

Connect with Jayson: LinkedIn / Twitter

GitLab security engineer Jayson Salazar

Tell us what you do here at GitLab:

I work as a security engineer on our Security Operations team. We work around the clock providing technical and procedural feedback, improving our security capabilities, interfacing amongst diverse stakeholders and responding to incidents to keep GitLab — the company, its employees and all its products — secure.

What’s the most challenging or rewarding aspect of your role?

I believe that one cannot understand that which cannot be easily defined and located and furthermore, that one cannot secure that which isn’t understood. In short, visibility is everything, at both small and large scales and, in my opinion, every security engineer ought to have a picture of the environment that they are trying to protect that is as accurate and detailed as possible.

Therefore, upon joining GitLab, I immediately tried to build a full-fledged mental map that bundled together the technologies, systems, ancillary artifacts and people with knowledge of them that GitLab leverages in daily operations. What I thought would be an easy, and rather uneventful task proved to be much harder to accomplish than expected as the days, weeks and months progressed.

Considering how diverse GitLab’s technological stack is and how many moving parts it has given that we’re all-remote, multi-cloud, SaaS, open-source and 800 employees strong; building such a mental scheme in one sitting was definitely overly ambitious. As time has progressed, however, I've come to terms with the idea that my understanding of GitLab as a whole; including technical aspects, as well as our values and culture, would continue to improve and cement itself and that it wasn’t a trivial task I could assign a deadline to or rush along. As of today, I’m very comfortable working with and reasoning through the different moving parts that make up GitLab, and getting to this point has been both very rewarding but also quite challenging.

And, what are the top 2-3 initiatives you’re currently focused on?

On the engineering side of my role, I’m focusing on architecting and implementing tools that improve our detection capabilities as a whole by allowing us to ingest, aggregate and build analysis and alerting pipelines around diverse and very interesting data sources. I’ve always been in love with data, hoarding it, slicing it, visualizing it and drilling down into it. By doing this we, the Security Operations team, create powerful tools that our teams rely on to spot, track and address security issues faster.

On the less glamorous front, I am quite passionate about (as everyone on our Security Operations team is) improving our processes, documentation and providing feedback on technical issues that I care deeply about. Therefore, you’ll often find me raising issues related to the security of our different products, or their components, as well as dealing with accrued technical debt, contributing to our Handbook or creating both technical and procedural documentation that other GitLab employees can rely on.

How did you get into security?

As a teenager, The Matrix was my favorite movie. The idea that rules and systems all around us existed for us to circumvent them really fascinated me. I gravitated towards “coding” because I wanted to recreate the weird unintelligible green terminal output shown on the screens of the Nebuchadnezzar. While in high-school some brief VBA and Excel explorations led me to Flash and Python. Before I knew it, I was spending my weekends during my freshman year in University frustrated but engaged playing wargames such as Over The Wire/ and Smash the Stack. It was during that time that I started seriously considering a career in information security. Although I went on to explore other areas both professionally and academically, such as software development and data analysis, which to this day I still quite enjoy, I was always drawn back to security.

What is the most significant piece of security advice you could provide to a colleague or friend?

Question yourself and your abilities, always within reason and, as long as you can deal with the emotional pressure. You can, and will be, wrong. When that happens, having countermeasures in place that you put there because you assumed your judgement could have been wrong is going to help you and your team greatly.

As with any industry, professionals working in cybersecurity can become rusty and comfortable with their day-to-day work. One incident comes after the other, every design decision becomes the same, using TLS, salt and hash, using a proper authentication and authorization scheme, buzzwords here buzzwords there, magical-security box from provider X or Y will save us, and on and on. All of the sudden, best practices become dogmas, rules of thumb turn into mental barriers, generous budgets devolve into excuses for lack of architectural work and the cybersecurity professional has, single-handedly, killed his ability to do meaningful, impactful, truly interesting and creative work. That’s a big one in my opinion. Another is being careful with burnout, practice self-care and don’t become cynical. You’re in cybersecurity because you care, you don’t need to be a rockstar to contribute, and yes, what you do matters.

From the perspective of your role, what’s GitLab doing better than anyone else in terms of security?

As an organization, we’ve quickly realized that, for security issues originating in artifacts that can be tracked and managed as code, it’s best to start looking for security issues early in the development process, before they materialize and carry real consequences, and not wait until the whole thing has been shipped.

GitLab’s Secure Team is working on creating and improving features that help teams mitigate security-related problems in their codebases before they occur and can be discovered via traditional means. In my opinion, this is a very interesting and powerful mindset-shift, we’re going from “number of bugs discovered” to “number of bugs prevented”.

What do you look forward to the most in security in the next 5 years?

To be honest, I’m not very thrilled about our collective future when it comes to cybersecurity. I believe some people greatly underestimate the complexity we’re facing while trying to secure the systems we’re building nowadays and this will become even more apparent in the next few years. It’s as if many companies are attempting to re-build their figurative planes mid-flight and that has the potential to backfire badly and affect customers and us all as a society; as it already has often in the past few years. However, I’m becoming increasingly optimistic as we’re seeing companies build out or empower their security teams to become more involved in the design and implementation phases of their infrastructure and, if applicable, their deliverables.

Is there an area of security research you think deserves more attention? Why?

Security analytics, and everything related to security analytics. Securing the internet for everyone little by little requires situational awareness, one of the best ways to get that is via data, lots of data. Said data will have to be gathered, stored, analysed and the related insights need to be shared. Privacy concerns aside, of which there are plenty, I’d like to see governments and public institutions gathering data about the number of systems they’ve updated in the last year, month or day, their patch levels, stacks they rely on, vulnerabilities they have fixed and much more. Imagine being able to rate the cybersecurity posture of a country as BB+ or AAA and aligning a nation’s (and by proxy its economy’s) cybersecurity efforts with financial success? Granted, this is just a random shower-thought I’ve had for a while but I think more research into “large scale security analytics and governance” could be an interesting endeavor.

What mainstream or industry propagated security myth would you like to be better understood?

That all companies should migrate to the “cloud”, or leverage IaaS or PaaS providers to operate, because having your crown jewels “up there” is intrinsically more secure. Of course, I’m not advocating for sticking to routines of the old days where spinning up servers meant having metal boxes on-premise. After all, I do work at GitLab and believe in the way we have adopted agility and in the many merits of DevOps. However, it’s crucial to acknowledge that the skills and mindsets required to properly secure traditional computing environments are, in many cases, radically different to those needed to operate secure cloud environments. Therefore, I think companies, especially small- and medium-sized companies without the budget to call-in experts once problems arise, should carefully plan the terms on which they want to migrate on-premise systems to the common IaaS providers or data centers with similar offerings. Ultimately, I’d like to see companies putting more emphasis in training their workforce properly before setting migration processes in motion that could potentially increase their existing technical and security debt.

Now, for the questions you really want to have answered:

What’s your favorite season?

Winter, hands-down. Cold weather, clear skies, the anticipation of Christmas season, snow, meeting friends for coffee and fireplaces, what’s there not to like?

What is that one food, you cannot live without?

Korean cuisine, especially Bulgogi. If the world ever ends, let it be with me eating Bulgogi as the sun sets.

When you’re not working, what do you enjoy doing/how do you spend your free time?

I quite enjoy discussing politics and social developments, listening to electronic music and watching and discussing deep, and not-so-deep, movies. Blade Runner, V for Vendetta, Matrix, Ghost in the Shell, The Girl with the Dragon Tattoo, and 50 First Dates are all favorites of mine.

On the creative side of things, I really enjoy writing poems. The way they touch people and how they interpret them in ways I could have never anticipated. It’s also a hobby that has become more and more enjoyable the more I share it with others, both in person and online.

Have a favorite quote?

I have many favorite quotes, but not really one I can call a core tenet of my personal philosophy or that drives inspiration. There is, however, a poem by William Ernst Henley that I often share, discuss with friends, think about, and always find myself reading again, and again: Invictus.

Photo by Akshay Nanavati on Unsplash

“What are the challenges and rewards of working in security at a cloud-native, all-remote company like GitLab? We ask @jdsalaro.” – Heather Simpson

Click to tweet

Edit this page View source