What’s it like working to secure one of the most transparent organizations in the world? To be a security practitioner in a highly iterative and agile environment? What does that look like and what kind of people thrive in that environment? It takes a certain individual... curious, analytical, collaborative and dedicated. Of course, there’s more than meets the eye when it comes to our GitLab Security team; they also tackle the hard topics like the age-old 'Is a hotdog a sandwich?' debate, Vim vs Emacs, and Linux distros.
We take securing the GitLab product and service and protecting our company very seriously. But, we try not to take ourselves too seriously. We hope you learn something new in this series, but that you enjoy yourself too.
Name: Roger Ostrander
Title: Senior security engineer, Abuse Operations
How long have you been at GitLab? I started on Mar. 26, 2018
GitLab handle: @rostrander
Connect with Roger: LinkedIn / Twitter
Tell us what you do here at GitLab:
I kill spam, I kill Bitcoin mining, I kill phishing and malware. If it’s bad, I kill it. And this isn’t just removal; I create automated tools that let us detect all these things and stop them ahead of time.
What’s the most challenging or rewarding aspect of your role?
I’m up against everyone in the world out to make a quick buck by spamming, which means it’s an arms race. I improve my detection, they find another way in, I detect that, they respond, etc. It’s definitely a challenge, and the reward is, of course, when I get to just completely wipe out a ton of spam or prevent it from being created in the first place. Who hasn’t seen spam and thought to themselves, “I wish I could do something about that right now”? I can!
And, what are the top 2-3 initiatives you’re currently focused on?
-
Snippet spam is currently a big problem, where people will put spammy search terms in our snippets hoping that search engines will pick up on them. Recently, there was an API change that’ll make that a lot easier to deal with.
-
Similarly, people create groups with names like “Watch this free movie online,” which are not only spam but also tend to be vectors for malware. So anyone who searches for “watch
free” hoping to pirate a film instead gets a link to a big, heaping pile of keylogging. Keeping on top of that is an ongoing priority, because of course there’s plenty of money to be made by taking over someone’s computer.
How did you get into security?
It started when I interviewed at Reddit nearly a decade ago – at first, I thought it was for an ordinary backend web development position. Then, halfway through the interview when I was talking to the CEO, he said “We’re actually more interested in your machine learning background, to fight spam.” Ever been in a job interview when you realize you’ve been interviewing for the wrong thing the entire time? I had to change gears pretty quickly, but it worked and I’ve been busting up bad actors ever since.
In the past decade, how has your area of expertise changed?
Quite a bit – I started out as a general backend web programmer, got into machine learning and spamfighting, where I learned quite a bit of frontend technology – even if I am terrible at the actual design work that generally accompanies that.
But the biggest influence on the breadth of my expertise came from a job that wasn’t actually a programming job at all: It was a role as a solutions architect for a NoSQL database company. If that sounds strangely vague to you, then you have some idea of what I did. It was a technical sales position where I’d fly on-site to various customers (some of them household names) and help them set up our product. The sheer amount of “big picture” experience I got from that was invaluable. When you’re programming, it’s very easy to fall into the specific area that you’re working on. Even if you do have backend experience, it’s hard to get a full idea of how something’s rolled out across a whole company, possibly worldwide. So, the sheer scope of that role allowed me to get a more complete view of how an entire system ought to work at the largest possible scale.
Tell us about a time when you failed professionally. How did you recover and what did you learn?
I once banned the entire front page of Reddit. We were dealing with issues similar to what I’m dealing with today, the “watch free movies” kind of spam/malware scenario I described above, and I’d noticed a pattern: Spammers would create a subreddit of their own and populate it with spam, for SEO purposes. So I created a processing script to find that behavior and made a list of all the subreddits they’d posted in and naively assumed they’d only posted to their own. I made a list, but it had several hundred items on it, so I spot checked them and it seemed everything was okay.
Surprise, it wasn’t! They’d posted in pretty much every popular subreddit, meaning my script banned high-profile, high-traffic subreddits. Also this was during the company’s all-hands so every single person in the company was asking “What happened to movies?” My response, of course, was a very calm, “I’M WORKING ON IT!” What I learned from that one was to fully check my results instead of simply spot checking, and that keeping logs of what your destructive scripts have done is mandatory.
GitLab is very unique in that we strive to be incredibly transparent... about everything. What sort of challenges does that present to you as a security professional? What opportunities?
This is an enormous issue for me, because while I am in the security department my area of expertise is anti-abuse. So, for example, if a vulnerability is reported to security, it won’t initially be public. Generally it’s only made public while it’s fixed. But when I come up with a new tactic for fighting spam… I don’t ever want that to be public! Even, perhaps especially, if it’s a simple tactic. If spammers knew specifically how they were being detected, they’d change their behavior accordingly. So it’s a very difficult balancing act.
Even so, there are opportunities – the snippets API feature, for example, came about because someone outside of GitLab requested it. They wanted to use it for anti-spam purposes just like I did, but the API doesn’t do any spam checking on its own. So that got to be developed in full view with all the benefits transparent development brings, but without giving away any secrets.
Now, for the questions you really want to have answered:
Vim or Emacs?
I learned VI(M) long ago as a practical necessity, and I highly recommend it. Every Unix system everywhere is going to have at least VI on it as a minimum, so if you know how to work with that then you can get something done no matter where you are. Emacs used to be my go-to “IDE”-type editor, but nowadays I generally use more specialized IDEs.
Is a hotdog a sandwich?
My wife works for the USDA, so she has opinions on this. Legally backed opinions, as it happens. Frankfurters are specifically quoted in policy as a “sandwich type product.” Citation: United States Department of Agriculture, "Food Standards and Labeling Policy Book." And I’m wise enough to agree with my wife. And also the law, I guess.
Is a taco a sandwich?
A taco is just a tacoid in the category of endofoodtors. What’s the problem?
Gif or Gif? (Gif or Jif?)
Look at that pronunciation guide right there in the question. One of those is spelled exactly like the word whose pronunciation is being debated. Just saying.
