Finding signal in unstructured data with Jose Nazario

Jose Nazario is a Senior Technical Director with the Mandiant Threat Intelligence team, where he focuses on how the organization is collecting intelligence, processing it, and distilling it into insights for their customers. Jose’s customers are decision-makers of major corporations and governments, who use Mandiant’s CyberThreat intel as part of their decision-making process to inform tactical and strategic decisions. 

Jose has worked in research and product roles and has played a key role in bringing multiple cutting-edge technologies to market. Throughout his career, Jose has focused on finding signal in unstructured data and using that to drive insights. Which is really what Product Strategy is all about. He’s also a brilliant technologist with multiple patents to his name and a fantastic human.

My favorite takeaways from this interview:

• Learning about how DARPA works and how closely tied it has been to tech innovation
• The Censys attack surface management tool started with a PDF report
• It doesn’t matter how useful your product is if you can’t get buy-in from the whole team
• Research and product are tightly tied together, but there’s a different skill set required for cutting edge research and taking a product to market
• How to give sales and product a common language

Here is our edited interview with Jose Nazario. 

AA: Can you tell me a little bit about your early days at Arbor Networks and how you ended up moving into a research and product role?

JN:  Yeah. A lot of the original foundational research from the University of Michigan that led into what is now Arbor, came out of the Defense Advanced Research Projects Agency, DARPA. The founder had this hypothesis that we could understand all sorts of broad internet activity by just listening to the internet and its unused portion. Being there, I got very fortunate and got to start analyzing that data and writing monthly reports. When you do DARPA projects, you always check in with your funding agency and provide monthly status reports to justify what you’re doing and give them part of the deliverables. So I didn’t build the system, but I got to analyze the data. 

Then, a few years later, the contract ran out, and we wound up replicating a core part of the project in what became ATLAS, or Arbor ATLAS. Arbor ATLAS was this idea of throwing sensors around smaller parts of the internet that were unused; listening, analyzing it, and producing reports. 

We launched the portal around 2006 at RSA. It was a live site, and I was involved in the whole project, from conception through delivery. By the time I left Arbor in 2012, I had added in DDOS attack tracking, botnet tracking, and malware analysis data. 

We were producing reports for our customers on this data as well as protecting the internet. We were in a position where someone could say “Help, we think we’re under attack,” and we could actually dig and find out what botnet was behind it.

But the business had no interest at all in commercializing it. I had a number of customers, like I said, asking ‘Where is this on the roadmap? What are you guys going to do with this? We love this thing. Could you guys do X, Y and Z with it?” and the business was like, “No, we ship boxes. That is the security service. We ship boxes, how does this fit into that?” 

I remember one of the last meetings I took was with somebody who asked for me by name, so I flew out to Boston to take the meeting and ran them through all these conversations of what it was, how it worked, what it let us do. And at the end of this spiel, the woman in charge of this group that was visiting us, she says “How Much? There’s a blank check on the table, how much is this thing?” The Vice President next to me says “It’s not for sale”, and I’m thinking “You fool. The world is changing, this is obvious to everybody but us.”

And it was really that experience - creating something new that made our customers happy, addressed unmet market needs, and the challenges that I faced in getting it to market - that really shaped what I did post-Arbor. 

AA: So, what were the business goals of all that research that you all had done, if it wasn’t to ultimately commercialize any of this work?

JN: So, a couple different things. One was of course to provide insight into the changing nature of the threats and making sure our products were up to date on what the attacks looked like. 

Also, it was driving a tremendous marketing value. We would look at what are called “Share of Voice” reports on a quarterly basis, and what our press looked like as compared to our competitors. We were easily 50% of the marketplace, or voice I should say, for the space of attacks. The business loved this. It was a small group, 5 or 6 people in my part of the org and indirectly another 20 or so. 

They saw that aspect of it, but there was never any interest in turning this into a customer-facing product or platform. They were constantly thinking, “How do we ship blue boxes? How do we keep that real estate at a data center? How do we compete with Juniper, Cisco, etc.?” 

So SaaS, Security as a Service, was completely alien to them. How would you make money in this space? How would you staff and operate it? What do the buyers look like? All of those market questions that I learned much later on are really, really crucial, we were not asking back then.

AA: Did you move into government-facing roles immediately after Arbor, or was that later?

JN: After Arbor, I moved into government-facing roles as a contractor, so I was never an official government employee. For me, it was almost a no-brainer. When I first got into the security industry in the late 90s, it was an almost Cambrian explosion of ideas and technologies. But at the time, I had almost become jaded with the security industry. I felt we were all chasing the same kinds of problems. 

It’s cyclical, there is some sort of explosion and then some sort of consolidation, and follow-on work. At the time, there was tremendous interest in network security and an early application of cryptographic signing, for authorizing applications and stuff like this. Again, technologies that we now take for granted, at the time, were radically being introduced. So 10 years on, I’m like, where are the breakthroughs?

AA: Do you feel that we’re back there now or do you think that there are breakthroughs happening right now?

JN: I think there are some interesting breakthroughs happening, I’m not keeping as strong a beat on them as I used to. Probably, again, because I’m jaded by the whole blockchain, crypto-world BS that has allowed a bunch of crap in the room.

I think there are some interesting things going on, especially in regards to “How do we work with people for moving through security incidents faster?” 

Again, that goes back to some of the original work I did to find that signal and noise. I’ve always been strapped for people and for resources, so I just automate the heck out of everything with really crappy code.     

AA: What are some of the challenges of developing new tools in the government-facing world vs. the private world?

JN: After Arbor, I felt jaded, and I really wanted to be part of the creation of the future of cybersecurity. I also really wanted to have an impact that mattered, but I was too old to pick up a rifle.

So I was at the initial research space, where their whole mandate is to discover what the unmet problems are, and help us meet them. 

You’re trying to find those unmet opportunities, and how they align with strengths that you can bring to the space. 

There's a lot of paperwork involved and it's taxpayer dollars so they want to make sure they're being spent wisely. So it's a little bit tedious in that regard, but it's also very exciting because you're basically inventing the future. 

Probably the biggest and most foundational challenge was not those motions, but rather staying ahead of things. 

I've been involved in a project, and we developed some technologies which got open sourced, called DARPA ICAS, and I think we presented on this at CSAW Threads in like 2014 or so. It was a really, really cool project and it was really exciting to build, but almost immediately we had a commercial vendor saying, “Oh well, we're already doing this.” 

That becomes one of the biggest challenges: to stay ahead of the marketplace. How do you identify all the right challenges that need to be addressed and stay not two, but actually 10 years ahead of where things are going to be?  

AA: Do you think it's possible to stay that far ahead? Because, generally, humans are pretty bad at predicting the future, right?

JN: I think there's a Bill Gates quote, “Things happen both faster and slower than you expect”, or something to that effect. It's very, very true; but at the same time, if you look at some of the technologies that DARPA has funded, the world they were trying to create has come to fruition in many respects. 

Look at Siri, look at mobile technologies, networking technologies. 20-25 years ago, we were talking about ubiquitous computing, and how we were gonna be carrying laptops around and wearables. 

The world didn’t quite happen like that, but we have cell phones that give us networking, location, time, photo and video capture. All these things we thought we wanted and all this technology sits either on our wrists or in our pockets.

So, I think they are able to predict the future and create it and influence it.

AA: Is DARPA making small individual bets or are they having to operate like a VC and bet on a lot of different paths at the same time and hope that some of them will succeed?

JN: I've heard it once described as the world’s largest venture capital fund. 

Again, remember, these are not frivolous investments. They're thoughtful, but they also go in knowing they're going to miss a lot more than they hit. So they structure it such that you are shooting for the stars, but if  you miss, you still  got to the moon. That, right there, is awesome.

AA: When did you start bringing products to market, and what were some skills or tools you identified along the way that were critical for getting something from an idea or research to the market?

JN: In my Arbor days, we built things for ourselves because no one else had them and found out they were valuable to others. So I got very fortunate, and sort of stumbled into it.

On the side, years before I built myself an early sort of Google News Prototype. There was Google News Alpha at the time or Beta, and I taught myself the basics of information retrieval and text mining by building my own news aggregator/distiller. Everyone is talking about this today: let's put these social media stories in front of you and organize them this way so they lose their repetition, but also use the repetition as a signal about what's important. 

After my contracting days, I was at Fastly and I got to take our Web Application Firewall to market. It was underway when I got there, so I got to complete it and see a mature organization taking a product to market from the periphery.

I think I actually underestimated how peripheral I was when I went to Censys and was their first product leader over there. That's really where I got an insight into getting products to market.

AA: What were some things you realized you didn’t know when you got to Censys?

JN: I think it was the crystal clarity of what the product role is and how it fits into the organization. I'm a slow learner; I had a nebulous definition of what it was and how it plays with the rest of the organization. 

If you look at it from a product manager’s or product management organization’s perspective, the number of different stakeholders you have is staggering. Basically, your job is to make sure the business is building what the market wants to buy. But you have to work with sales and customer success and marketing and finance and all these other groups as well to ask, OK, can the business profitably deliver this? 

This is where my skills are really, really terrible. 

Where I’m strongest is in the more technical aspects. Inventing the future and proving out the market interest, both through competitive analysis as well as taking early prototypes and betas to customers for that initial feedback. Understanding where the pain points are, what the hardest thing customers have to do in a day is, where they spend all their time, and so on. 

At Censys, for example, we built the attack surface management product. I got there and pretty quickly we had some working prototype, and we could take it to some of our existing customers. 

So one of the things you want to have is a stable group of customers who know, love, and trust you, and are willing to give you honest feedback. Hopefully they're paying customers, and hopefully they have a lot of overlap with the market you want to get into. They're not the top 1%, but they're indicators of where the market is and where it's going to be and can give you that brutal feedback. 

We took them this really crummy early prototype, I think it was like a 400-page PDF. We basically took all our different data tables and charts and just spat them into a gigantic PDF. My objective was to be able to get to the customer to answer two questions. First of all, are we solving a problem you have that nothing else solves for you right now, and you’re willing to pay for this hopefully? And secondly, are we accurate? Are we close with this? Is it useful?

I remember sitting in a meeting, the customer is flipping through this prototype on the call, and he's like, “Ugh. OK. We all know all this stuff. Hold on a second, that shouldn't be here.” We hear his keyboard tapping. “Oh crap, I gotta file a ticket.” It was immediate validation. We helped this large organization basically prevent another breach. That was the moment for me that A, I knew that we were on the right track, and B, I thought, “how do we make this happen immediately for everybody?”

As we get into the actual motions of sales and finance, that's where my skills are really, really blind. At a smaller company like Censys, those were massive, glaring holes. But at an organization like FireEye/Mandiant those are fine because we have other people who can come in and close those gaps. 

For example, I recently helped transition the next version of a product from R&D into the engineering organization. Being able to design and implement and lead a team to build a product and validate it in the market: that's where my strengths are the strongest. Now, as we’re looking to take this to market in the coming months and quarters, that's where my skills are terrible and we brought somebody else in to take it over.

It's exciting to watch it in the hands of masters, people that are really skilled at this.

AA: You've talked about how your early research was largely in finding signal in unstructured data. What are some of the gotchas that people can run into when prototyping a new product where they can get distracted by the noise?

JN: There's two or three things I try to bring to it. 

The first is: don't set the bar too high. A lot of engineers set the bar very high because they're thinking “Well, this has got to scale. This has got to support 10,000 simultaneous users.” Well, it doesn’t. You’re showing this off to one person. Just make it not crash. That's it. Set the bar super low so you can get there quickly. 

The second: have a lot of conversations. If you have something you can quickly show to people, you can set up a ton of conversations, listen and take a lot of notes. 

The third: repeat this cycle often. By keeping the prototype small, you can come back in a couple of days or weeks and have new conversations with people. Again, it's not falling prey to the sunk cost fallacy. It's not a slow turnaround. Nope: you’re back in a week with some new ideas. 

A lot of this comes back to Teresa Torres’ work on Continuous Discovery, where the team - product, engineering, and design - are constantly exploring and discovering market gaps and how they might address them. 

A tool I built for us at Censys was inspired in part by the Heilmeir Catechism, which tries to capture and evaluate innovative ideas. Heilmeier was one of the most consequential DARPA directors, and the questions he posed - and that I used in this tool at Censys - were things like “how does the user do this now? If they could do this, what could they do next? What have you done to validate the feasibility?” Questions like that, we put them into a Confluence template. So rather than a wasteland of a Jira backlog of illegible tickets expressing feature requests, we were able to capture use cases, how important it was to what customers - is this a nice to have or a must have - and any prototyping we had performed. This gave product and sales a common vocabulary, “hey Jose, I think we have another customer who would want X.” And when you begin to dig into it, it’s a rough set of requirements that’s got multiple contributors and believers behind it. 

AA: What are some of the ways you broaden your horizons? You said one, take a lot of cups of coffee with people outside your field. Are there any other things you do to try to get outside of that bubble?

JN: So, when I left Arbor I had a conversation with the Dug Song. He's like, “you're deep in a rut man, you just don't see it.” He was so right. Dug is right about a lot of things. 

I was feeling like “UGH. This has all been done before.” 

Now anytime I find myself thinking that way, “This has all been done before”, I consciously stop, and say “Hold on a second, what might be new here?” And I listen and look.

It winds up being a very valuable exercise to always be going through “What am I missing here?” This idea seems wacky, what am I missing? And it's hard. I'm not always successful at it, but when I am, it's been rewarded.