What a Site Reliability Engineer Really Does…in DevOps

We really, really build ourselves into a corner with the internet and mobile and cloud and Agile “at scale”. Good news is, we’re engineers that can invent ourselves out of anything, or at least that’s what’s made all this money so far.

What Is a Site Reliability Engineer?

Srsly. Wikipedia. Too lazy? Fine, from Wikipedia (please donate):

Site Reliability Engineering (SRE) is a discipline that incorporates aspects of software engineering and applies that to IT operations problems. The main goals are to create ultra-scalable and highly reliable software systems. According to Ben Treynor, founder of Google’s Site Reliability Team, SRE is “what happens when a software engineer is tasked with what used to be called operations.”[1]

What kind of this ninja trickery is this? Using common sense to make learn how to hire the best people in technology? Why would Google spill the beans on this hiring secret? Maybe they’re sick of dealing with our broken shit.

Our digital systems are ALL distributed and complex now. How can we still expect that having some ignorant code-jockey in a cubicle who never uses what they make control the entire business with the stroke of a keyboard? Because: we are cost-accounting brainwashed and forget that the job to do needs the right experience and skill to do it well. Meanwhile, we keep under-hiring operations and over-hire developers such that there’s a 1-to-who-knows ratio between the people that press one button and the people that press another.

If You’re Offended By What I’ve Described, Congratulations!

I am too. Things that are so complex no one person can understand them, those things are dangerous. Banking apps that aren’t secure, mapping apps that get us lost and late, social media apps that show our kids their first porn, CGM devices that cost more in maintenance fees than their worth…it offends me when these things don’t work. Technology that works is how I make sure I have money for a family, sponsored, biological, or otherwise.

Our tech industry should be hiring people that can comprehend the things they deliver. People pay for things that work. If you don’t care about others, at least you’ll care about making money, and “right” software in a customer-obsessed market makes the most money.

It’s particularly offensive when the hybrid phoenix of a job title that ‘Site Reliability Engineer’ embodies goes largely unnoticed in high tech corporate mindsets. “What the hell is that, your latest professional title advancement scheme? Just because you mashed these words together doesn’t mean you deserve a raise!!!” If you know the following things, you deserve a salary that rivals an enterprise VP of marketing:

  • What your software should do
  • How your software does what it does
  • How to communicate the value of the things you’re working on
  • Don’t mind being woken up when it’s broken for someone
  • Ignore those around you that don’t think the above is relevant to do their jobs

Go forth and make your first salary million in a few years, y’all who can. Do this well and grow.

Why Do We See Site Reliability Engineering on the Rise?

The tech industry is now at the point where we completely forgot that the persons who build software should know how to operate that software when it other people depend on it. Big money, consumer insatiability, customer centricity, and digital transformation has skyrocketed the imperative to make the modern enterprise business engine their engineering teams. We build shiny, complicated, and highly profitable things. What did we expect?

We, the nerds, lured jocks in with our shiny things such as the Altair, BBS, and the entire mobile revolution…and they brought their friends. CFOs, ‘professional CEOs’, and other people that look at a hoodie like its pajamas that violate the corporate dress code. We allowed things to get this way #waterfall #agile #WomenInTech by being egotistical, lazy, impatient, and unkind. These are our chickens coming home to roost.

And now we have to reinvent a way out of the ‘shallow engineering’ tech culture that looks skeptically at #DevOps as a management problem. I don’t mean that everyone on your engineering team has to code, but the people who do code should understand the impact of what they do. This is ethical and this is practical. This is how you make your next billions.

This is the new horizon for impactful, profitable, and scalable tech culture:

On #InternationalWomensDay, I guess that is all for now.


FUEL for Your Brain: On Focus, Usefulness, Execution, Learning

I hate acronyms. My dad used to use them far too much, the kind of guy that was more smart in retrospect than the kind of boy I was understood at the time. Kind, thoughtful, quiet, and invested in people around him.

My current thought product is an acronym, “FUEL”, based on a few key practices that I find are valuable to my current line of work as a Change Agent. These practices take time to develop and are only truly useful when used in parallel.

  • Focus: ability to right-size activities to close the task(s) currently underway
  • Usefulness: ability to gauge effectiveness of work and reprioritize based on new ideas/objectives/activities
  • Execution: ability to match skill to task, collaborate the plan, and resolve blockers as they arise
  • Learning: ability to observe outcomes and refactor them into useful ways to improve all of the above

Real-time Example of FUEL

Last night after a meetup, I had a beer with someone I’d met before at a local conference but hadn’t dived into. The opportunity presented itself, so I stayed a little later than I normally would. They are a CTO for a 50-person startup in town. Net-net:

Paul: “What’s weighing on you right now man, work related?” (L)
Them: “Kind of glad someone asked…we have people issues.” (E)
Paul: “You’re not alone…what kind?” (E/F)
Them: “There are a few ‘senior’ engineers that don’t produce like others.” (U)
Paul: “What’s your plan for them and the rest of your team?” (U)
Them: “We just laid off one after giving him a path, but the other two, I don’t know…maybe add metrics, visibility…they’re kind of SPOFs.” (U)
Paul: “…so you can quantify what you already know? How did we arrive here?” (U/L)
Them: “They were here at the beginning, hence ‘senior’, but one guy hasn’t committed code since Sept (5 months)!” (E)
Paul: “Got it, they don’t ‘git’ it. [laughs] How are you and the leadership team  helping to coach other junior engineers?” (L)
Them: “Well that’s the problem maybe. We don’t exactly have a culture yet, but our C-level relationships with each other are solid.” (U/E)
Paul: “I once heard that great leaders define their success by fostering other leaders. Do you know who your real ‘senior’ engineers are?” (F/L)
Them: “Well I kind of already know who deserves the chance to step up.” (E)
Paul: “That’s good, but not enough. People often hesitate on new things simply because they haven’t experienced how it works yet. Your coaching needs to help those people get over any blockers to proving one way or the other if they can do the job well/right/better.” (FUEL)
Them: “I’m going to talk to our CEO about this. Can I get your card?”
Paul: “Only if you intend to use it.”

Good enough exercise and learning for me for one night.


Technical Recruitment 101 – Advocates vs. Evangelists

What’s the difference between a technical evangelist and an advocate? What are these terms even? If you’re a technical recruiter, I encourage you to know the answer to all of these questions by reading this piece for the next 4mins.

What is a “Technical Evangelist”?

A way to simplify the definition of an evangelist is to onboard people to their particular topic, do whatever it takes (hackathons, sponsorships, contribution, enablement sessions, flights…lots and lots of flights…blogging, interviews, code samples, research, t-shirt design, customer support, server setup). They wear all the hats, usually at the same time.

What is an “Advocate”?

An advocate defines their job based on the success of the customer (or anyone really) at the job they most need to do. Deep listening, directed dialog, metrics extraction, change impact quantification, being kind, and work framing are all tools in an advocate’s toolbelt. It’s caring about the other person first, then of course yourself secondly.

Advocacy vs. Evangelism

Advocates are good facilitators. Unlike evangelists, they don’t assume that because a hat needs a wearer, that it must be them to wear it. They see the whole field, not just the ball in-front of them. Advocates identify what needs to be done cross-functionally and help to match people who want to and can do a thing to get that done. They think about the problem that people are trying to solve, and put resources into motion to reach that goal.

Naturally, when I frame advocacy this way, it’s easy to see why evangelism leads to burnout. Advocacy leads to faster burndown (e.g. sprint burndown charts) because facilitation, a scalability reinforcer, is at the core of advocacy. What problem are you trying to solve? What’s required to get there? What challenges will arise? How can we most effectively help each other?

How to Qualify Who Should Apply For Which Role?

If someone demonstrates that they can differentiate between facilitating needs over taking on things as a personal responsibility to accomplish, you may just have the potential for an advocate. If a candidate lists off all the things they do without a clear definition of why first, you’re better off funneling them into an evangelist role so they can either learn a better balance or burn out and find something else.

Advocates build plans and often play a significant role in driving those plans to completion. Internal, external, independent, or consultative. When they do take ownership over a goal, you know that it will get closed, even if it wasn’t achieved to expectation, it will get closed and retro’d.

In Practice, I Say “Advocacy over Evangelism”

Evangelists talk about their product and rarely take full responsibility for anything. They are often driven by others to do things, go places, speak under sponsored time, build samples, and be engaged with customers. It’s right in the job title. “Evangelism” in Latin can be translated as “messenger”. In other words, they have been told to deliver a message. Usually the messenger has nothing to do with the crafting process of the message, and would you want someone whose job is to talk rather than listen involved in what your message to people should say? I wouldn’t.

Put another way, an evangelist mindset attempts to frame the problems of the world into terms that their product can solve. Their focus is the product that they’re incentivized to proliferate. If it doesn’t quite fit a customer’s situation, too bad, we’ll make it fit. I don’t know the job you’re trying to accomplish because I didn’t bother to ask, but I’m going to offer you a random solution anyway. And naturally, introducing a wrong-fit solution produces negative consequences (like lost time, lost money, and lost opportunity).

An advocacy mindset exfoliates information required to make decisions about how to best accomplish a customer’s true goal. Your product is not your software. Your product is a right-fit of people, process, and technology for a customer to successfully accomplish their goal. Your overlap to their situation in strictly wheelhouse terms might only by 10% or just one specific job. But if you can understand how important (or not) what you do is to completing the job they have at hand, then you can quickly fit what you offer to their needs.

This is the heart of the way product teams should “go to market”. They proceed as explorers and caretakers, not “disruptors” simply because it sounds cool. In this world, caring about how to help someone with their job better every day is disruptive. It’s honestly disruptive, to sales, to marketing, to product management, and to vision holders in technology.

The NRA, Magazines, and Video Games

I think we forget that we have trained a generation of children to commit mass shootings. “Video Trainings” like Call of Duty and Battlefield should be declassified as “video games” because killing is not a game. Likewise, we should re-sensitize to how alluring the symbol of a gun is to adults who need to feel powerful.

NRA and Magazines

Yesterday, a neighborhood friend posted a photo of gun magazines in epic display at one of our local chain pharmacies. After mulling over what this meant for about this for 2mins, I responded:

PaulSBruce: "Hey @cvspharmacy and @Walgreens, what kind of user experience design decision allows vendor contracts to dictate that children should see this? you need some ethical reverse engineering." based on neighbor's recent Facebook post of a photo of a full, foot-level rack of gun-related magazines. Neighbor says: "Eye level with my 2 year old at a pharmacy in liberal Massachusetts".

Granted, this is a liberal-as-fuck move, using social media to push against a brand publicly to take ownership for businesses and corporate contractual obligations. Thank you, yes it is.

If you’ve ever really worked with a kid, you know that getting down to eye level with them really helps to engage and motivate them. So stuff like this at their eye level is bound to engage and motivate them,

What really concerned me is that there’s a population of adults now who are apparently consuming this shit enough for there to be a rampant market for it…in the most liberal state in the United States. Either this anti-pattern was creeping up on us over years, or there’s some serious money being pumped into this retrograde mindset that proposes the 2nd amendment meant for AR-15s to an inalienable right.

I highly doubt it was a rogue decision by a single branch manager because, having flown to Minnesota and Wisconsin this week, I saw with my own eyes that this was a reoccurring trend, same thing in Detroit. Like a signal to people who would use these kinds of fire arms that they should stock up quickly. A reminder that you can only safe if you pack more than the next guy, that enough bullets will solve any problem. I can’t even imagine what it’s like in the southern states right now, maybe they’re gearing up to stage a Confederate overthrow.

It’s the money that bothers me. To get this many separate magazine titles on a local rack, magazine not gun surprisingly, there have to be some serious contracts in place. This goes along with what the manager told a community member when confronted by the senselessness of this:

So vendors (of the magazines) must approve a change to floor placement in pharmacy chains? Sounds awfully like a well-known contractual obligation to me. Why would a gun magazine retail distribution contract stipulate that all gun magazines must be placed on the bottom rack? You must be 18 in order to become a legal consumer of a gun. Why wouldn’t the magazines want to promote at eye level of their legal consumers?

“Mommy, Daddy, why don’t we own a gun?” is a powerful thing for a parent to hear. And because you can get gun loans with no background checks in some states, practically anyone can rapidly resolve that question in lethal force.

I think the problem here is multi-level. Brands like Delta, United, MetLife, Bank of Omaha, Enterprise, Windham, and soon others publicly denounce the NRA while for-now-president Donald Trump and goons like Wayne LaPierre equally denounce gun control efforts. Figureheads aren’t innocuous, they embolden an already primed generation. And on that…

NRA and Video Games

When I was a kid, video games were a cursed luxury. I paid for my own Famicom…I mean, Nintendo Entertainment System…and games too. Repairs even, when I spilled cereal into the thing (don’t ask me to reconstruct that memory please).

My conservative Christian parents begrudgingly lent me scant amounts of time on the family TV, that is until someone had the bright idea to put the NES on a small black and white TV that my recently passed Grampy had left to me. Spoiler: it was me. I am an engineer and things just occur to me sometimes. And I wanted to beat every goal that some big kid who made the game set for me. Inside there, I could do great things, certainly better than out in real life.

In my ‘Xellenial’ generation, most parents either didn’t understand video games or otherwise weren’t the type to care about much else. I made sure that in my house, video games would be a family activity. The one’s we chose are puzzle-solving, narration read requiring, fun incentivizing pieces of amazing art by teams of people who love what they do. While some people struggle to get their kids to care about books at all, ours read so quickly through everything we bring home from the library that they create their own stories, which are more than often very interesting.

My partner and I have chosen to be very intentional parents. We feed our kids the best we can afford. We limit screen times to the weekends. We don’t subscribe to religion. We are kind and sincere people. We will never give our kids phones because they will pay for them if it’s important enough to them.

So it’s easy for me to view video games as a good thing in my house. That’s not so kids who have been given technology and no constraints. The habit is formed to scratch the itch to get what you want, to win, to watch the next one, until it’s done. Unconstrained screen time encourages obsessive-compulsive personalities in kids.

Please just go to a GameStop and look at all the highest grossing titles. They’re mostly first-person shooters like Call of Duty, Battlefield, Destiny 2, Wolfenstein, Sniper, Rising Storm, and get this…”Get Even”, a story who’s heroine has to shoot her way out of dungeons, office buildings, and SCHOOLS.

Killing is not a game. Interactive killing stories are not video “games”, they are early video “trainings”. Take away the killing, like in Mario Odyssey, where multi-player and assist mode help various ages in a family all progress through problem-solving challenges together, and you have an actual video “game”.

[Note: listening to the Super Mario Odyssey background soundtrack, it’s really great to hear that the musical director took the time to create an 8-bit version of the 44k version for the switch between 3D and 2D gameplay. That’s like a little gift for me as a parent who grew up with these themes.]

Titles that include violence should be barred from sale as “video games” and be regulated under federal law the same as movies and other adult forms of entertainment. To these video game vendors, you make blood money and you should be eradicated along with the mental disease you spread.

“I would not squelch legislative efforts to deal with what is perceived by some to be a significant and developing social problem. If differently framed statutes are enacted by the States or by the Federal Government, we can consider the constitutionality of those laws when cases challenging them are presented to us” says U.S. Supreme Court Justice Samuel Alito. [ref]

A “Developing Social Problem”

Is gun violence really a ‘developing social problem’? Well, let’s consider two facts: 1) if someone shoots someone else in public, that’s a social problem and 2) gun violence is definitely developing.

Gun Violence Statistics by State between 2012 and 2016

There are other statistics that indicate that gun sales are going up while homicides by gun is going down. The margin variance in units per home is often plotted against a seemingly dramatic drop in incidents of violence. The type of violence matters if we consider what kids have been “trained” to do: pick up a weapon, find a way out of a building, kill what gets in your way. The mentally ill underage in this country have an easy onramp to becoming a statistic more like this:

There have been more than 1,600 mass shootings since Sandy Hook

Also look at: https://www.cnn.com/2016/06/13/health/mass-shootings-in-america-in-charts-and-graphs-trnd/index.html

“The practices and beliefs of the founding generation establish that ‘the freedom of speech,’ as originally understood, does not include a right to speak to minors (or a right of minors to access speech) without going through the minors’ parents or guardians,” says Justices Clarence Thomas.

That’s right, so when major pharmacies branded chains with managers that are members of my local community contractually decide to allow the NRA to engage with my children about owning a gun, when families allow children to play shooting games without limits, and when we don’t confront these situations without hesitation when they arise, this is what creates a fatal national epidemic.

What do we do about it?

I drove to my pharmacies, grocery stores, and convenience stores to look for these displays. Find a few, tell them about the social movement and how they don’t want that massive bad PR and inevitable sales dip next month due to boycott, and sometimes things change:

Still at eye level to a 2 year old, but at least the surface area for little eyes is reduced. I’d rather have Rachel Ray glaring at them than the muzzle of an AR-15.

Local acts, local impacts.

DevOps Testing Strategy for Dummies

Testing in a DevOps culture is very different from traditional QA scenarios. I talk to all kinds of teams, from Fortune 100 to startups, all on the journey to adapt and innovate. What happens to testers in this new world?

This article is a bundle of content related to why and how software teams can align and improve their testing strategy. I addresses “right fit” to across org and process, cost center vs. value stream, and many other dynamics in testing culture.

“No Testing Strategy, No DevOps”
by Paul Bruce
on CapitalOne Dev Exchange

“Testing Strategy in DevOps”

Greater Boston Selenium Users Group
on February 15th 2018
at Salesforce Commerce Cloud HQ

Soundcloud: background on
“a looming crisis in QA”

Setting Up Your Own Selenium Grid on AWS

This article describes how DevOps teams can quickly spin up a reliable, cost-effective Selenium grid for automated testing in minutes.

What Is Selenium & Selenium Grid?

If you’ve never heard of Selenium before, climb out from under that rock.

Selenium is a test automation framework that lets you script interactions against your web apps and run them in popular web browsers. Selenium Grid is a management service that coordinates the use of many instances of these browsers. You can point your Selenium scripts to a Grid, specify which “capabilities” (OS version, browser version, etc.) it would like in an interactive session, then drive actions in that browser session to verify functional correctness of a target app or website.

This is what it typically looks like in Selenium code:

Pretty simple, really, for what’s really going on under all the layers of code and HTTP protocol communications.

Why Use a AWS for Your Selenium Grid

There are plenty of services that host Selenium-compliant grids, such as Sauce Labs and Browser Stack, and I recommend you consider one of the many SaaS options if you don’t want to maintain and upgrade a grid on your own. However, if your team has reasonable automation skills and the human resources to do so, the benefit of a DIY grid is complete control for a fraction of the cost associated with hosted services.

Though I abhor the idea of paying hundreds of dollars per month for only a few parallel browser instances, a word of warning about DIY grids over SaaS alternatives:

  • You don’t get same-day support for the latest browsers and OS versions
  • You don’t benefit from script-to-driver version independence
  • You don’t get infinite scalability without some serious effort
  • You don’t get 24/7 operational oversight unless you pay someone to do it

If you (and your team) are okay with the above disclaimers and simply just want a quick, cheap way to spin up a grid, then the steps below suffice:

  1. Launch a Ubuntu linux distro to act as the Docker host.
    I used an t2.xlarge at $0.19/hr to get 4vCPU variable and 16GB memory. Also set SDA storage to 32gb.
  2. Log in to your instance via SSH
  3. Switch to root account to install Docker with ‘sudo su’ command
  4. Install cAdvisor with this command:
  5. Install Selenium Hub:
  6. Create containers for your various browsers using debug versions and unique external port numbers (typically 5901, 5902, etc.):
  7. In the security group you created/assigned during EC2 instance setup, add the following inbound rules:
    7777 (cAdvisor)
    4444 (Selenium Hub)
    5900-6000 (VNC for various browsers)
  8. Verify that your grid contains the browsers you spun up:
  9. Connect to VNC remote services on instances using the ports assigned in the above docker run commands. On a Mac, I recommend VNC Viewer.
  10. Reference your grid in a Selenium script, using port 4444:

[picture of code]

Now when you run your script(s), particularly in parallel, you should see nodes on your grid running your automated script.

How Is a Selenium Grid Better Than Local Execution?

Well, for one, it’s not popping up multiple browsers on your workstation while you’re trying to get other work done. In build and continuous integration scenarios, you need to run automated Selenium scripts on an environment that doesn’t include the laptops your developers take home with them at night.

You also want your test environment to be as versioned and reliable as possible, so having an on-demand grid that can be scripted as an AWS YAML descriptor and that meets your manual, on-demand, and CI testing requirements is a modern must.

Why Did I Write This Tutorial?

I’m a sometimes performance engineer. I needed a disposable Selenium Grid to show how to use a subset of browser sessions during a large load test to collect real-user experience metrics in NeoLoad. I didn’t want to pay hundreds of dollars for a monthly subscription to something I could build in about 30mins.

Why am I running a subset of real browsers as part of a load test? That, my friends, is the really interesting part that I discuss in a later post.

Recap of DevOps Days Boston 2017 with David Fredricks

This week, I had the opportunity to continue a conversation started weeks ago with David Fredricks, organizer of DevOps Days Boston.

You can watch it on YouTube or listen via Soundcloud. Transcript below.

The Incredible Impact of Open Spaces

Paul: All right so welcome everyone my name is Paul Bruce and once again I’m back here with a member of the DevOps community, Dave Fredericks. Now Dave you organize the DevOps Boston event is that correct?

Dave: Yeah that’s correct. I’ve participated as a volunteer organizer for the last three years, involved for the last four.

Paul: Excellent…and I got a chance to meet you beforehand, I think at one of the DevOps Days Boston meetups, but then also we got to chat at the event and it was a really good event. I think a number of different things were really just cohesed really well, particularly from my point of view, the collaborative open spaces. Can you tell us a little bit about what that’s like how that got into the conference schedule?

Dave: Yes, certainly. So open spaces is a really interesting kind of platform that’s unique to DevOps Days. Basically how it works is everybody comes up with topics during the event after listening to some of the keynotes. Some discussions that are interesting to individuals, a lot of times you want to add on your personal perspective into, not only offering new ideas or maybe even some suggestions, but asking specific questions.

A way of being able to do that is by getting everyone together at the event over common topics. You basically vote on different topics that are of interest to you and they can actually go anywhere from cultural to personal to technology as a whole.

The idea is, there’s a few rules, it’s basically:

  1. what’s being said is what needs to be said
  2. who’s there is the people that need to be there
  3. when it starts and when it ends is the time it starts and ends

Those are the only kind of guidelines that we go by and the idea is to get people who usually wouldn’t be open to public speaking to be able to have a chance and an opportunity to either share some ideas, ask questions specifically and directly to different individuals and to have an open forum.

The real values that come out of it are real specific dialogue, the biggest thing is new introductions and relationships that are created.

The hope is that throughout the year after the event, a DevOps stage event is for you to be able to get contact information of individuals who are in the same space, at the same stage as yourself to have an outside outlet to be able to bounce ideas off of through the year as you start to face some of the challenges as you as an engineer try to solve problems.

Paul: Yeah that was one of those things that really clicked for me, being part of a number of those open spaces, I saw exactly what you said which was people were far more likely to comment and to share and ask questions. And in a larger audience and I think the other element of that is the fact that not only do they get the share but they get instant feedback.

And this is one of those core tenants I think of DevOps, in my mind, is this concept of continuous learning. But you don’t learn unless you know what’s going on and you don’t know what’s going on unless you [as an organization] radiate information which is typically facilitated by feedback loops. So whether we’re talking about technology feedback loops or real people feedback loops, I think that’s really helpful.

So can I back up for a second and ask you a slightly broader question about DevOps: in your mind how would you define DevOps?

What Is DevOps, Really?

Dave: Great question. You know these this is one that in our community we talk a lot about, especially for folks who are outside of quote-unquote “DevOps thought process”, knowing that it’s something that’s taking off as a force in the software world.

One of the things we do is to talk about how do we define DevOps. The biggest thing for me is DevOps means different things to different people and it’s all about context and perspective, where you come from and where you’ve been and what challenges you’re trying to solve. So when I meet somebody new who’s in this space and they’re starting to kind of either chant or evangelize to me without first getting a baseline perspective as to where I’m at and what I was doing and what I’m trying to solve, immediately has me question, “okay, are you trying to push your ideals down on me?”,

This is what DevOps means to me: getting folks to work together in an efficient collaborative manner to solve a common goal, period.

It has nothing to do with tools. It has nothing to do with process. It has nothing to do with frameworks. It’s all about getting people together, teaching context, having empathy, understanding what somebody’s doing, why they need to do it, and what what they’ve been doing in the past. You share your ways of doing it and then together when you have a sense of “okay, I know why this person has to do things, I know the reason why they’re thinking this way”, you can efficiently solve problems and for me that’s that’s what DevOps is to the core, right there.

Paul: So one one thing I heard from that is it starts with people, right? It doesn’t start with tools, it doesn’t start with how you’ve been doing it; it starts with people and really understanding the context and the perspective that they bring to the table. Is that right?

Dave: Yeah, Paul, you you nailed it right there. It starts, it continues, and it ends with people. Ultimately I take the concepts and the core principles of DevOps, and you can apply that to any industry, any product, any delivery, any manufacturing, and it really is bringing people together to work more efficiently to solve a common problem.

What Is DevOps Not?

Paul: And so actually people are doing that, you’ll hear the prevalence of these amalgam terms like DevSecOps, DevTestQAOps. And I kind of take issue with that in the sense that I understand how important terminology and clear labels for things. As a practitioner and engineer, as soon as somebody starts to blow out a term to mean “all the things”, my red flags get raised up instantly.

That doesn’t mean that [DevOps] doesn’t include other people, but can you tell us a little bit about how important the scope is of DevOps to you? And just kind of following that up with some context, I was able to speak to Ken Mugrage from the DevOps Days Seattle, and he was very clear about how if we blow it out into all the things, “DevOps” loses its value.

And so I put this to you: why is a pantheistic term, if DevOps grows to that, why is that a problem?

Dave: No, that’s a great thought. I want to take this back a little bit to identify why are all these actions added on, how and why this is how [DevOps] is being branded in this way. This was a discussion that I have, especially with growing teams.

One of the biggest things I talked about with organizations is, first and foremost, technically there is no DevOps engineers. So why label it that way?

There’s No Such Thing as a “DevOps Engineer”

When I started working with a lot more enterprises, I helped organizations transform their development to be much more modern so that they can have quicker release cycles and feedback. It’s one of the things that used to frustrate me, was like “hey, we need five DevOps engineers!”. That doesn’t mean anything to me, you got to explain on a day to day basis, what is this person doing, and ultimately, why are you labeling these folks as DevOps engineers?

And I I had some interesting feedback which came from the product marketing side. They were like, “Dave, we’re in the enterprise. We’re used to big long deploys of software in order to get it to our customers, and a lot of times we don’t know if our customers are even getting any value out of what we’re producing. When we’re releasing every year and waiting for six months to get the actual feedback from our customers, it doesn’t make any sense.”

So you see this large swath of folks trying to get into this space to build software quicker to have faster feedback to be able to add more value to end users.

These individuals don’t really understand this whole open source community, they don’t understand how the strength of the community is really the value.

“So we don’t know how to really market. We don’t know how to communicate to the group in a way for us to be able to blanket it all together. So we just scoped it into this thing and we call it #DevOps and everything gets that kind of label to it.”

From my experience what I’m starting to see is a lot more of these organizations who are specific to security, to testing, in a way of being able to catch and grasp that member of the audience, it’s “let’s throw it in, Dev and Sec Ops, Dev Quality Ops. What starts to happen in my mind and what I’m what I’m worried about is that people start to lose the real purpose.

Paul: So basically the exact same thing that happened to Agile. Everybody forgot to have agility as one of the core tenants that people check in on, on a regular basis such that they internalize that, and that is where their activities and their tools flow from, right?

Dave: Yes exactly. If you start to get too focused on the terminologies and the labeling of things and forget the context as to why you’re practicing it, ultimately the further down stream you get and the more generations that start to get folded into the process, they’ll start to lose the actual scope, “hey we’re trying to get people to to work together in a more collaborative manner to be efficient and to be able to deliver quickly.”

How to Be a Good DevOps (Citizen, Vendor, Employer)

Paul: Yeah, one thing that I did recently was put out an article (and thank you you, you had shared it to a number of people and I think that’s half the reason why I got some attention). It was essentially how to be a good DevOps vendor. It took the approach of looking at it from the customers perspective. The implementation of that was over a simplified customer journey and then chronologically through that journey, I went through and basically made statements from an outsider’s perspective onto different groups whether it be product, marketing, sales.

Back to your perspective, I get that it has to fundamentally start with people because people are what build teams and teams are what build software and software is what affects us. But the team affects us and individuals affect us, and so it does make sense to keep that as a core of value, to consider personal responsibility and also the responsibility of the team to have these cultural aspects present.

But unfortunately I think what happens is that we do need tools and you know, conferences are notorious for needing some kind of funding and becoming self-funding is really hard, and so out comes sponsor packages and I mean it’s an ecosystem. All software is eventually, in most people’s minds, going to make money and so this is where I was coming from, understanding that there is no such thing as a DevOps vendor or a DevOps tool or a DevOps job/position. Yet the fact is that when you’re closely aligned with the thinking of another person and “DevOps” is the term they’re using, it’s easy for these vendors to kind of bring that in and pull that into their messaging.

So I guess my my point of view on that is that we are gonna have to deal with that but it’s kind of a constant battle against the pantheism of trying to “all the things” a term [DevOps] but in the meantime we also do have to represent those tenants to more than just the developers and operations. If you really want to sell to developers and operations or teams that are looking, or they have internalized DevOps, they’re going to be looking at the world from this interesting perspective. And they’ll be looking across the tool chain to figure out who sounds like they’re blowing smoke up [you know where].

If a tool vendor or a service provider does not understand the core of DevOps, then their messaging, their selling process, their product ideation…it’s all not going to jive with the real market.

After after a recent Boston DevOps meetup we dove into this for what like an hour and a half, and just really talked about how do we actually do this. My concern is that when we start to move this into the enterprise (and by the way, the good principles of DevOps should be moveable to the enterprise, right? If they work, they work, and it’s a matter of fitting to context) that I think, while the core of it is culture, we can’t just live in this sort of kumbaya world.

We really have to figure out how to scale DevOps principals up and out into the enterprise setting so that, by the way, these good principles have a positive impact on things like automated insurance, things like machine learning in terms of healthcare, defense and government settings.

So I’m working on that on the side but in the meantime, what do you think about scaling to the enterprise? What does that even mean for DevOps?

How DevOps Is Re-writing Management Decisions

Dave: Yeah, that’s a great point. It’s an interesting challenge. There’s a lot of organizations who are facing it. Right now, I’m dealing with situations where we’re starting to see a lot of enterprise buy instead of trying to build it themselves. One thing they have is capital and resources. So the idea is, “if we don’t know or we can’t make it, it’s the bye versus build, like why go out and try to do what people already are being really successful in doing in something that we don’t understand too well? Let’s just go ahead and absorb some of these startups…”

Paul: Do you mean actually purchasing startups in order to just fill that technical gap in an organization? So I don’t want to name names, but I’m thinking of a very large enterprise that just recently bought up one of the most well-known API monitoring services out there, and people are freaking out like “oh gosh, what’s going to happen, are they going to de-culture this awesome group of guys and gals?”

Dave: I’m dealing with the same thing within an organization, a large security company buying a smaller more nimble security product with a lot of open source options. They’re putting out there trying to create groundswell to get this tool for free into the hands of engineers, let them play with it so they can understand how it works and create some kind of a swell within the engineering teams and then we’ll come up to the top start talking to the executives about, “Hey, what challenges are you facing in this broad space?”, where you’re trying to protect not only year your customers information but also information about your company.

As they start to have that type of dialogue, all of a sudden the executives within the organization starts to look down, talking to their engineering group and saying “hey, what do you know, what have you played with, what do you think is interesting, how do you think we should be solving this problem?” You’ve already created that initial lift of inertia in engineering, then they say hey let’s go with this product…we already know how it works, we’ve been you tooling around with it. Win-win, right?

So this is a completely different way of thinking of how enterprises used to be selling products into their customers. It was always a top-down approach…let’s talk to the executives who have the purchase power, float it down and then they’ll disseminate that information in the way that we roll it out into the engineering team. That’s how you could do it in the old-school way. Now in today’s new world, a lot of tools are available for you to play with for free and when enterprise organizations start to try to come into this space, they’re really kind of blindsided by this whole new content creation process.

Selling Into DevOps Takes Understanding DevOps

What I’m starting to see is they’re at least now recognizing we do not know how to sell to this to this community of this group. We know we really want to get into the space, we want to do it the right way, what do we do right and you know to your point with your article, I’ve shared your article with all of the enterprises that I’ve have been talking to me about this problem because I can’t teach them about the thought process of open source.

I mean, we can look back in the 60s, the MIT days, where the two groups kind of split off. A lot of us in the DevOps space already have the mentality of like “hey, you know we want to be able to share a lot of this stuff but we do want value for hard work we do”. But for the most part there’s different ways of doing it versus everything is being paid for with the enterprise mentality.

What I’m starting to experience is there’s a lot of organizations out there that are realizing it’s exponential value once they start to get into this community and…

the brand loyalty within the DevOps community is tremendous

…but the challenge that is in front of us right now is really the learnings piece and I’m thinking it’s a leadership issue (this is my own personal view). It’s enterprise leadership that needs to get out of the way and allow for new blood to come in to be able to understand the kind of movement. I’ve been doing a little, as much as I can to try to influence old leadership. It’s a challenge and a lot of it has to do with success syndrome. You’ve been doing it in certain way for decades. It’s a great case study that we’re gonna be able to kind of sit back and watch in the next five years

Calling All Researchers: Inclusion Means You Too!

Paul: Yeah and you know, there’s so much going on, no one person can do it alone. So without plugging any commercial products of any kind (that’s not my motion) I have started something called the iterativeresearch.org, which is essentially a bunch of contributors to research. As they go along, it could be lightweight contributions, simply just pocketing articles and getting into a feed of people who pay attention, it’s writers too, but the point is it’s not on a brand that’s connected to a pay-for services. And you know I would love, for this conversation to really start flowing in that direction because I think it takes many perspectives, right?

The core of this is it’s an inclusive conversation, not an exclusive one.

So understanding that you are a busy man and we’re at the top of our time, are there one or two things that you want to give a shout-out to or any particular resources that people can go to, events, communities, open-source forums, anything like that?

Get Out to a DevOps Tribe Near You!

Dave: Yeah, you know, thank you so much for the opportunity first and foremost, we’re gonna have to do it again! One of the things I really would highly recommend to folks who are interested in getting more involved, start to look at some local meetups that you have going on. There are some great folks within every community in whatever city, whatever small town, who are interested in sharing ideas and in thoughts in challenges. All you have to do is get out there and look. Go find your tribe! The biggest thing is don’t sit back and wait and sit on your hands and expect for interest to come to you.

The whole constant learner, the Kaizen mentality, be better tomorrow than you are today, be better today than you are yesterday. It lives and dies in DevOps and the way to do it is start to talk to folks who you’re not used to talking to.

Don’t be afraid get out there introduce yourself and have a good time. Life is learning.

Paul: Cool. So that’s David Frederick’s everyone and thank you David for spending the time with me. Do you prefer going by David, Dave?

Dave: Dave, David, either way.

Paul: Dave/David, I’ve really enjoyed it was great being able to spend some time. We’ll circle back. Thank you so much! Cheers!


More from DevOps Days Boston:



Folding Open Source into Enterprise DevOps

Open source software (OSS) is a foundational part of the modern software delivery lifecycle. Enterprise teams with DevOps aspirations face unique challenges in compliance, security, reliability, and sustainability of OSS components. Organizations-in-transformation must have a complete picture of risk when integrating open source components.

This article explores how to continuously factor in community and ecosystem health into OSS risk analysis strategy.

The Acquisition Process for Open Source Software

Developers need to build on the successes and contributions of others. Having the flexibility to integrate new open source components and new versions of existing dependencies enables teams to go fast, but external code must be checked and validated before becoming part of the trusted stack.

Including someone else’s software is an important moment of engagement. Enterprises typically wrap a formal ‘acquisition’ process around this process, where the ‘supplier’ (the entity who provides the software/service) and the ‘acquirer’ (the entity who wants to integrate the software/service) contractualize.

Though there are already commercial approaches to introducing software packages safely by companies like Sonatype, Black Duck, and others, my question extends beyond the tools conversation to encompass the longer-term picture of identifying and managing risk in software delivery.

Enterprises care deeply about risk. Without addressing this concern, engineering teams will never actualize the benefits of DevOps.

This is a tangible application of the need for DevOps to not only apply at an individual team level, but in the broader organization as well. It takes alignment between a team who needs software and teams providing compliance and legal services to all do so in an expedient manner that matches the clock speed of software delivery.

Communities Empower Enterprises to Address this Gap

Today in a Global Open Source Governance Group Chat, I asked the question:

“What are some methods for determining how significant a supplier/vendor OSS and community contributions are, relative to acquirer confidence?”

This question stems from my involvement with the IEEE 2675 working group, particularly because I see:

  • Prolific use of OSS use in DevOps and in enterprise contexts
  • Reluctance and concern (rightly so) around integration of OSS in enterprise software development and operation in production
  • The convergence of compliance and automation considerations
  • How important transparency and collaboration is to the health of OSS, but also to the supply and acquisition processes in a DevOps lifecycle

The group, expertly facilitated by Lauri Apple, also included key insights from Paul Burt and Jamie Allen. A log of the conversation can be found on Twitter.

As open source projects (like Swagger/OADF for instance) become increasingly important to enterprise software delivery, health and ecosystem tracking also becomes equally important to any new components being introduced.

My point-of-view is that organizations should prepare a checklist for software teams to construct a complete picture of risk introduced by OSS (not to mention proprietary) components. This checklist must include not only static analysis metrics but support, engagement, funding, and contribution considerations.

Measuring OSS Project + Community Health

The group had many suggestions that I wouldn’t have otherwise thought about, another reason for more people getting involved in dialogs like this.

There are already providers of aggregate information on open source community health and contribution metrics such as CHAOSS, a Linux Foundation project, and Bitergia. This data can be integrated easily into dependency management scripts in Groovy, npm, Ant, Maven, etc. and at the very least written in to a delivery pipeline as part of pre-build validation (BVT is too late).

And there is honest, hard-hitting research on open source software…which you should take the time to read….from Nadia Eghbal under the Ford Foundation in a report called ‘Roads and Bridges: The Unseen Labor Behind Our Digital Infrastructure‘. If you don’t have time to read it, buy some text-to-speech software and listen to it when you’re in transit.

The group also identified some key characteristics of OSS community health not necessarily tracked by established services, such as:

  • Same day response on reported issues, even if it’s simply acknowledgement
  • PRs under the “magic number” of 400 lines of code…tends to be the limit for # of bugs and useful feedback
  • Outage response, sandbox availability
  • Distribution of component versions across multiple central repositories

More to Come…From YOU

As I integrate both my own learnings and other voices from the community into the larger Enterprise DevOps conversation, the one thing that will be missed is YOUR THOUGHTS, whether your in a large organization or simply in a small team.

Please share your thoughts in the comments section below! Or ping me up on Twitter@paulsbruce or LinkedIn.

More reading:

Recap of DevOps Days Boston 2017 with Corey Quinn

This weekend, I had the chance to have a ‘distributed beer’ with Corey Quinn of Last Week in AWS to chat about the DevOps Days Boston 2017 conference last week. We provide a few takeaways each in about 5 minutes.

You can watch it on Youtube and listen on Soundcloud.

My Recaps of Day 1:


Links from this chat:


Beyond DevOps: The ‘Next’ Management Theory

In a conversation today with Ken Mugrage (organizer of DevOps Days Seattle), the scope of the term ‘DevOps’ came up enough to purposely double-click into it.

‘DevOps’ Is (and Should Be) Limited In Scope

Ken’s view is that the primary context for DevOps is in terms of culture, as opposed to processes, practices, or tools. To me, that’s fine, but there’s so much not accounted for that I feel I have to generalize a bit to get to where I’m comfortable parsing the hydra of topics in the space.

Like M-theory which attempts to draw relationships in how fundamental particles interact with each other, I think that DevOps is just a single view of a particular facet of the technology management gem.

DevOps is an implementation of a more general theory, a ‘next’ mindset over managing the hydra. DevOps addresses how developers and operations can more cohesively function together. Injecting all-the-things is counter to the scope of DevOps.

Zen-in: A New Management Theory for Everyone

Zen-in (ぜんいん[全員]) is a Japanese term that means ‘everyone in the group’. It infers a boundary, but challenges you to think of who is inside that boundary. Is it you? Is it not them? Why not? Who decides? Why?

By ‘management’ theory, I don’t mean another ‘the silo of management’. I literally mean the need to manage complexity, personal, technological, and organizational. Abstracting up a bit, the general principals of this theory are:

  • Convergence (groups come together to accomplish a necessarily shared goal)
  • Inclusion (all parties have a voice, acceptance of constraints)
  • Focus (alignment and optimization of goal, strategies, and tactics)
  • Improvement (learning loops, resultant actions, measurement, skills, acceleration, workforce refactoring, effective recruiting)
  • Actualization (self-management, cultural equilibrium, personal fulfillment)

I’ll be writing more on this moving forward as I explore each of these concepts, but for now I think I’ve found a basic framework that covers a lot of territory.

I Need Your Help to Evolve This Conversation

True to Zen-in, if you’re reading this, you’re already in the ‘group’. Your opinions, questions, and perspectives are necessary to iterate over how these concepts fit together.

Share thoughts in the comments section below! Or ping me up on Twitter@paulsbruce or LinkedIn.