A performance engineer’s job is to get things to work really, really well.
Some might say that the difference between being a performance tester and a performance engineer boils down to scope. The scope of a tester is testing, to construct, execute and verify test results. An engineer seeks to understand, validate, and improve the operational context of a system.
Sure, let’s go with that for now, but really the difference is an appetite for curiosity. Some people treat monoliths as something to fear or control. Others explore them, learn how to move beyond them, and how to bring others along in the journey.
Testing Is Just a Necessary Tactic of an Engineer
Imagine being an advisor to a professional musician, their performance engineer. What would that involve? You wouldn’t just administer tests, you would carefully coach, craft instruction, listen and observe, seek counsel from other musicians and advisors, ultimately to provide the best possible path forward to your client. You would need to know their domain, their processes, their talents and weaknesses, their struggle.
With software teams and complex distributed systems, a lot can go wrong very quickly. Everyone tends to assume their best intentions manifest into their code, that what they build is today’s best. Then time goes by and everything more than 6 months old is already brownfield. What if the design of a thing is already so riddled with false assumptions and unknowns that everything is brownfield before it even begins.
Pretend with me for a moment, that if you were to embody the software you write, become your code, and look at your operational lifecycle as if it was your binary career, your future would be a bleak landscape of retirement options. Your code has a half-life.
Everything Is Flawed from the Moment of Inception
Most software is like this…not complete shit but more like well-intentioned gift baskets full of fruits, candies, pretty things, easter eggs, and bunny droppings. Spoils the whole fucking lot when you find them in there. A session management microservice that only starts to lose sessions once a few hundred people are active. An obese 3mb CSS file accidentally included in the final deployment. A reindexing process that tanks your order fulfillment process to 45 seconds, giving customers just enough time to rethink.
Performance engineer doesn’t simply polish turds. We help people not to build broken systems to begin with. In planning meetings, we coach people to ask critical performance questions by asking those questions in a way that appeals to their ego and curiosity at a time that’s cost effective to do so. We write in BIG BOLD RED SHARPIE in a corner of the sprint board what the percentage slow-down to the login process the nightly build as now caused. We develop an easy way to assess the performance of changes and new code, so that task templates in JIRA can include the “performance checkbox” in a meaningful way with simple steps on a wiki page.
Engineers Ask Questions Because Curiosity Is Their Skill
We ask how a young SRE’s good intentions of wrapping u statistical R models from a data sciences product team in Docker containers to speed deployment to production will affect resources, how they intend on measuring the change impact so that the CFO isn’t going to be knocking down their door the next day.
We ask why the architects didn’t impose requirements on their GraphQL queries to deliver only the fields necessary within JSON responses to mobile app clients, so that developers aren’t even allowed to reinvent the ‘SELECT * FROM’ mistake so rampant in legacy relational and OLAP systems.
We ask what the appropriate limits should be to auto-scaling and load balancing strategies and when we’d like to be alerted that our instance limits and contractual bandwidth limits are approaching cutoff levels. We provide cross-domain expertise from Ops, Dev, and Test to continuously integrate the evidence of false assumptions back into the earliest cycle possible. There should be processes in place to expose and capture things which can’t always be known at the time of planning.
Testers ask questions (or should) before they start testing. Entry/exit criteria, requirements gathering, test data, branch coverage expectations, results format, sure. Testing is important but is only a tactic.
Engineers Improve Process, Systems, and Teams
In contrast, engineering has the curiosity and the expertise to get ahead of testing so that when it comes time, the only surprises are the ones that are actually surprising, those problems that no one could have anticipated, and to advise on how to solve them based on evidence and team feedbacks collected throughout planning, implementation, and operation cycles.
An engineer’s greatest hope is to make things work really, really well. That hope extends beyond the software, the hardware, and the environment. It includes the teams, the processes, the business risks, and the end-user expectations.
Foreward: Since I highly doubt the following concepts will see the light of day in the final draft of IEEE 2675, I wanted to document that in fact I pushed this to the group June 15th 2017. During a subsequent review, it got huge push-back from our Curator in Chief, the early first of a future string of events that lead me to publish this primary work on my personal blog.
What is a ‘value chain’?
A value chain is a set of activities that a firm operating in a specific industry performs in order to deliver a valuable product or service for the market. The concept comes through business management and was first described by Michael Porter in his 1985 publication, Competitive Advantage: Creating and Sustaining Superior Performance.
The idea of the value chain is based on the process view of organizations, the idea of seeing a manufacturing (or service) organization as a system, made up of subsystems each with inputs, transformation processes and outputs. Inputs, transformation processes, and outputs involve the acquisition and consumption of resources – money, labor, materials, equipment, buildings, land, administration and management. How value chain activities are carried out determines costs and affects profits.— IfM, Cambridge
As related to DevOps, the Value Chain for Software Delivery is an application of a lifecycle perspective that scopes standards adherence to only certain individuals based on their participation in primary or supporting activities related to a particular software product.
DevOps is about continuously delivering value to users/customers/consumers. A value chain perspective disaggregates activities from an organizational funnel, making it easier for teams and consumers of a particular product or service to ask “does this thing meet the standard” without unintentionally including other unrelated teams or products, were the question be phrased as “does this organization meet the standard”.
What problem are we solving by using ‘value chain’?
Adoption. In large organizations with many independent groups or product teams, DevOps principals may apply across the value chain of one product, but not another, provided these products are completely independent from one another. For an organization or team to claim that a particular software product adheres to this standard, all aspects of that product’s value chain must implement the principals and practices set forth in this standard.
Examples of where the broadness of using “organization” presents a challenge to adoption:
Consulting agencies with many independent project teams on working on separate contracts/products
If only one of those contacts require adherence to 2675, does this require every team/contract (both in the future and retroactively) to do the same?
Using “value chain” would scope 2675 adherence to any and all parties performing activities germane to delivering that specific contract/product
We know that if even one team implements DevOps per 2675 and sees success, organizations are likely to grow that out to other teams over time; “value chain” helps adoption of the standard.
Enterprises in the midst of transformation to DevOps
Can they claim adherence on a specific product if the “whole organization” can’t yet?
Much like the above agencies argument, when the scope of adherence is based on activities relating to delivery of a project, enterprises are far more capable of becoming “DevOps ready” because they can grow the practice out *over time*
In DevOps, key success factors are determined – driven – by customers, the end user.
Organization’s requirements from non-technical internal agencies
Can we expect that the legal or HR departments are also “DevOps”? This would of course need to be defined by activities that support the delivery side of the business (i.e. billing, purchasing, etc.), begetting an activities-based perspective on implementation over organizational labeling
Why is this of importance to IEEE 2675?
In layman’s terms, adoption of IEEE 2675 at an organizational level can’t happen overnight, especially in large enterprises with many teams. Fortunately, it doesn’t have to, provided we adequately scope ‘shall’ statements with a perspective that A) is reasonable in scope and impact on the org, and B) enables parties to agree on what it means for a software product to have been developed and delivered using this standard.
How and where would we use ‘value chain’?
Places in the text where use of the word ‘organization’ could infer that IEEE 2675 must be implemented across the whole organization before any single team or product could claim adherence to the standard. For instance:
“Organizations shall implement effective continuous delivery procedures aligned with the architecture definition in a manner that meets the business needs of the system.” … “DevOps itself requires effective architecture across the organization to ensure that application build, package and deployment procedures are implemented in a robust and consistent manner.” (6.4.4)
“Organizations shall implement effective continuous delivery procedures within a particular value chain that are aligned with the architecture definition and in a manner that meets the business needs of the system.” … “DevOps itself requires effective architecture across the whole value chain to ensure that application build, package and deployment procedures are implemented in a robust and consistent manner.”
“Organizations shall maintain an accurate record of both code deployed and fully automated mechanisms in order to remove obsolete code.”
“Organizations shall maintain an accurate record of both code deployed and fully automated mechanisms in a value chain in order to remove obsolete code.”
Curiosity is what drives engineers, and is equal parts curse and companion. An engineer isn’t limited to development or operations. An engineer would be a problem-solver in both areas, probably more. Curiosity is a surprisingly rare quality in people, even in technology.
If you want to know how something works, take it apart and observe. My first digital systems disassembly was a Sony Discman in 1989. Whatever I did, I fixed it. The feeling was powerful. It just took me 25 years to realize that there are many broken things in the world and to prioritize which one’s I involve myself with. Understanding the problem is crucial.
This is how I approach many conversations, navigating purposely and politely until there’s a useful reframing. People aren’t things, so be kind, be sensitive, and be patient. When you engage, learn about their biggest challenges, how they approach things, and what drives them. Just start with that.
[If my 80’s discman was still around, it would be like “yup, those were the days”. My 8086 XT clone next to it would splutter out some op codes. My Mega Man game watch would be waterlogged and stuck in a loop. Which all lead me to the next action…]
Put things back together again so that they work, hopefully, better than before. It’s just courtesy. In commit-worthy code that’s called hygiene. In conversation, that’s called maintaining a shared view or vision. Do these things enough and you’ll find that the way to a common goal is easy easier than with clutter obscurring your journey, and for others’. When you can row in the same direction, you get to your destination a whole lot faster.
Put it with other things to see where it doesn’t work. That’s integration and it’s not always easy, especially if it’s your own code or new auto-scaling configuration that causes unforeseen things to blow up. Get to know what your thing does before and after you put it out in the wild. Be honest with yourself and others about the time this takes.
There are some things you can take apart, and some things you can’t. If you can’t or if it’s too much effort for not enough value, move on to another learning tool, but remain committed to your goal. In the light of fundamental flaws in how we think about security, privacy, and basic human welfare right now, seek something you’re proud and grateful to do. [There are some very worthy things happening in Boston right now.]
Let’s be clear. I’m mostly writing this so that I don’t have to have another bullshit conversation with a high-level agency “technical” recruiter that doesn’t really know what the hell DevOps really is. But before you misinterpret what’s to come as a horn-rimmed trash session on recruiters (only a motif, I promise), consider that had you not read to the end, you would never have learned how to hire more efficiently, effectively, and ethically.
Boston, We Have a Big Problem
Aside from the occasional scuffle at a Java meetup, I’m in the Greater Boston Area, and Boston isn’t exactly known as a shining example of the tech boom. Sure, we have Facebook, Amazon, ZipCar, TripAdvisor, Chewy, RaizLabs, and Pivotal amongst others. We also have Oracle, IBM, Salesforce, and a plethora of other institutionalized madness from the Fortune 500 which typically drags our communal technical proficiency rating down year after year. We also have some of the most dedicated, seasoned professionals which you’d be hard-pressed to find in Silicon Valley during a fictional OSCON meets AWS Re:invent meets RSA all rolled into one.
Our problem is hiring. Everyone everywhere does not have the problem quite like we do. Its potency is not diminished in the DevOps space simply because its a generally applicable point to make for any highly skilled market. When something is as culturally intertwined as True DevOps mindset is in high performing teams, traditional recruiting approaches will always fall flat on their face. What people are calling DevOps positions right now are a loose collection of uninformed guesses, buzzwords, poorly crafted hiring pitches, philosophical paradoxes, voodoo, and outright corporate misunderstandings.
Recruiter, Know Thyself
What’s required is simply a sea change in recruiting mindset: the fastest way to qualify people is for you to qualify yourself. I don’t mean credentials, I mean the way you ‘smell’. Not by your morning shower and shave, but by what real practitioners need to hear from your mouths in the first 30 seconds: “I’ve really been thinking about how to right-fit you and a few of my clients,” or something equally real and challenging. How about “I’m interested to know what kind of work you’d like to be doing…” or “Which do you like better, people or code?” (that one is for all you recruiters looking to place the even more elusive ‘DevOps Manager’ position, which as it turns out is just a normal technical manager that’s passionate about coaching and improvement and customers and building future leaders.)
Or you could just leave us really good hires out of it, focus on the email overload flowing from opaque B2B recruiting firms currently choking your inbox (yes, I have recruiter friends and we talk), and then you’d be bidding for the lowest common denominators and margins, not the highest ones. Good luck with that approach, it leads to burnout and we DevOps people know a million ways to get burnt out. But unless you’re ethically fine with placing unqualified people with unqualified teams for a buck, the first option is better for you and for everyone you churn through week after week.
Be Transparent..to a Fault
Recruiters and hiring managers, there is nothing worthwhile to hide from someone like me that isn’t abundantly already transparent through what you don’t know about the organization you represent. I don’t have to ask questions about the org, just about you and your relationship with actual decision-makers, and what you don’t understand or know enough about to hire on behalf of True DevOps teams properly.
To get over cursory technical qualifications, ask people for examples of their work, or better yet look for them first; the simple act of a prospect answering you with examples of their expertise (or simply knowledge to-date) in particular areas is something unqualified people can’t do and unmotivated people simply won’t do. If your candidate hasn’t used any of a dozen or more social platforms (like Stack Overflow, Medium, LinkedIn, etc.) to publish their own stuff, encourage them to so you can pass it along to the real decision makers.
Once these minimum-viable qualification hoops are behind us, bring something to the table. Have a spine and a brain and a perspective about the challenge you’re looking for my help to solve in an org. Understand the broader issues the organization you’re representing is having, then ask us which ones we think we actually have the desire and a real shot at helping to move forward.
Suspend disbelief for a moment…
Drop the bullshit. Tell us who your client is when we ask, how many and which positions they have open, how many other candidates you’re playing off each other for some high-dollar plot of glory, and for god’s sake be prepared to describe your client’s *engineering culture* like it was your own family. Go to a meetup every so often, or better, help to organize one. Sit through and listen, stop emailing from your phone through the presentations. Absorb what’s happening, how people respond to certain topics, and if you don’t find yourself startled awake by clapping, you may just learn something.
Against all corporate recruiting conventional wisdom, when a potential candidate comes at you with such Maslowian questions as “Can you tell me a bit about their culture?” and “What challenges there would I even be interested in?”, definitely don’t talk about benefits packages (that are mostly all the same at a certain level anyway) and please don’t use things like “occupational environment”, “team synergy”, or “interpersonal skills”. Please be real, explain to us what you do so we can understand if you do it better than anyone else we will talk to that week, and leave the qualification script at the door.
You’re selling you to us first, then you earn the right to sell your client to us. If the order of those two things is in reverse, it’s equally transparent where your placement priorities lie.
Hiring Managers: Who’s Doing Things Differently?
Off the top of my head, I can think of a few folks in the local Boston DevOps meetup that are good examples of how to place highly qualified practitioners:
Dave Fredricks,Founder, eninjia.io
This guy is legit. Hard working, constantly advocating, and an early organizer of DevOps Days Boston, amongst other dedicated individuals.
Sam Oliver, 3yrs self-employed recruiter, now FTE at PathAI
This woman really rolls up her sleeves, as a co-organizer of the Boston DevOps meetups, and smartly carved out the “we’re hiring” pitches from the “let’s talk tech” conversation that this crowd is known to like as separate things. #listening
Kara Lehman, Principal Recruitment Consultant, Huxley
Her stock is climbing with me. We first chatted it up in 2017 and when I asked her what she was doing at the nerd-fest of a meetup at Pivotal, she said that she “needs to understand this thing called DevOps better”, which is far more enlightened of a response that about 99% of other local recruiters.
If your name isn’t in the above list, nothing personal, I’m writing this at 10pm on a Monday. Let’s have a conversation where I vet you out and then maybe I’ll write about you too. In the mean time, prepare yourself because it will be me who’s interviewing you.
Apologies and Thanks
If you got this far, my gift to you is my true gratefulness for feedback you may have and maybe a retroactive apology for saying things harshly. We need to cut through bullshit, especially the corporate flavors of it, and this is my personal blog anyway.
Recruiters: if you “can’t do these kinds of things” in the position you’re currently in, consider that you would probably earn far more by taking a new approach (like the one in this article) on your own and making 100% commission on your own closes instead of a measly 50k/year plus.
Also, you can also reach out to me via LinkedIn and let’s talk about your challenges in hiring, training, managing, or fostering a DevOps-minded team. I’m much nicer in person than in this post.
We really, really build ourselves into a corner with the internet and mobile and cloud and Agile “at scale”. Good news is, we’re engineers that can invent ourselves out of anything, or at least that’s what’s made all this money so far.
What Is a Site Reliability Engineer?
Srsly. Wikipedia. Too lazy? Fine, from Wikipedia (please donate):
Site Reliability Engineering (SRE) is a discipline that incorporates aspects of software engineering and applies that to IT operations problems. The main goals are to create ultra-scalable and highly reliable software systems. According to Ben Treynor, founder of Google’s Site Reliability Team, SRE is “what happens when a software engineer is tasked with what used to be called operations.”
What kind of this ninja trickery is this? Using common sense to make learn how to hire the best people in technology? Why would Google spill the beans on this hiring secret? Maybe they’re sick of dealing with our broken shit.
Our digital systems are ALL distributed and complex now. How can we still expect that having some ignorant code-jockey in a cubicle who never uses what they make control the entire business with the stroke of a keyboard? Because: we are cost-accounting brainwashed and forget that the job to do needs the right experience and skill to do it well. Meanwhile, we keep under-hiring operations and over-hire developers such that there’s a 1-to-who-knows ratio between the people that press one button and the people that press another.
If You’re Offended By What I’ve Described, Congratulations!
I am too. Things that are so complex no one person can understand them, those things are dangerous. Banking apps that aren’t secure, mapping apps that get us lost and late, social media apps that show our kids their first porn, CGM devices that cost more in maintenance fees than their worth…it offends me when these things don’t work. Technology that works is how I make sure I have money for a family, sponsored, biological, or otherwise.
Our tech industry should be hiring people that can comprehend the things they deliver. People pay for things that work. If you don’t care about others, at least you’ll care about making money, and “right” software in a customer-obsessed market makes the most money.
It’s particularly offensive when the hybrid phoenix of a job title that ‘Site Reliability Engineer’ embodies goes largely unnoticed in high tech corporate mindsets. “What the hell is that, your latest professional title advancement scheme? Just because you mashed these words together doesn’t mean you deserve a raise!!!” If you know the following things, you deserve a salary that rivals an enterprise VP of marketing:
What your software should do
How your software does what it does
How to communicate the value of the things you’re working on
Don’t mind being woken up when it’s broken for someone
Ignore those around you that don’t think the above is relevant to do their jobs
Go forth and make your first salary million in a few years, y’all who can. Do this well and grow.
Why Do We See Site Reliability Engineering on the Rise?
The tech industry is now at the point where we completely forgot that the persons who build software should know how to operate that software when it other people depend on it. Big money, consumer insatiability, customer centricity, and digital transformation has skyrocketed the imperative to make the modern enterprise business engine their engineering teams. We build shiny, complicated, and highly profitable things. What did we expect?
We, the nerds, lured jocks in with our shiny things such as the Altair, BBS, and the entire mobile revolution…and they brought their friends. CFOs, ‘professional CEOs’, and other people that look at a hoodie like its pajamas that violate the corporate dress code. We allowed things to get this way #waterfall #agile #WomenInTech by being egotistical, lazy, impatient, and unkind. These are our chickens coming home to roost.
And now we have to reinvent a way out of the ‘shallow engineering’ tech culture that looks skeptically at #DevOps as a management problem. I don’t mean that everyone on your engineering team has to code, but the people who do code should understand the impact of what they do. This is ethical and this is practical. This is how you make your next billions.
This is the new horizon for impactful, profitable, and scalable tech culture:
Testing in a DevOps culture is very different from traditional QA scenarios. I talk to all kinds of teams, from Fortune 100 to startups, all on the journey to adapt and innovate. What happens to testers in this new world?
This article is a bundle of content related to why and how software teams can align and improve their testing strategy. I addresses “right fit” to across org and process, cost center vs. value stream, and many other dynamics in testing culture.
Paul: All right so welcome everyone my name is Paul Bruce and once again I’m back here with a member of the DevOps community, Dave Fredericks. Now Dave you organize the DevOps Boston event is that correct?
Dave: Yeah that’s correct. I’ve participated as a volunteer organizer for the last three years, involved for the last four.
Paul: Excellent…and I got a chance to meet you beforehand, I think at one of the DevOps Days Boston meetups, but then also we got to chat at the event and it was a really good event. I think a number of different things were really just cohesed really well, particularly from my point of view, the collaborative open spaces. Can you tell us a little bit about what that’s like how that got into the conference schedule?
Dave: Yes, certainly. So open spaces is a really interesting kind of platform that’s unique to DevOps Days. Basically how it works is everybody comes up with topics during the event after listening to some of the keynotes. Some discussions that are interesting to individuals, a lot of times you want to add on your personal perspective into, not only offering new ideas or maybe even some suggestions, but asking specific questions.
A way of being able to do that is by getting everyone together at the event over common topics. You basically vote on different topics that are of interest to you and they can actually go anywhere from cultural to personal to technology as a whole.
The idea is, there’s a few rules, it’s basically:
what’s being said is what needs to be said
who’s there is the people that need to be there
when it starts and when it ends is the time it starts and ends
Those are the only kind of guidelines that we go by and the idea is to get people who usually wouldn’t be open to public speaking to be able to have a chance and an opportunity to either share some ideas, ask questions specifically and directly to different individuals and to have an open forum.
The real values that come out of it are real specific dialogue, the biggest thing is new introductions and relationships that are created.
The hope is that throughout the year after the event, a DevOps stage event is for you to be able to get contact information of individuals who are in the same space, at the same stage as yourself to have an outside outlet to be able to bounce ideas off of through the year as you start to face some of the challenges as you as an engineer try to solve problems.
Paul: Yeah that was one of those things that really clicked for me, being part of a number of those open spaces, I saw exactly what you said which was people were far more likely to comment and to share and ask questions. And in a larger audience and I think the other element of that is the fact that not only do they get the share but they get instant feedback.
And this is one of those core tenants I think of DevOps, in my mind, is this concept of continuous learning. But you don’t learn unless you know what’s going on and you don’t know what’s going on unless you [as an organization] radiate information which is typically facilitated by feedback loops. So whether we’re talking about technology feedback loops or real people feedback loops, I think that’s really helpful.
So can I back up for a second and ask you a slightly broader question about DevOps: in your mind how would you define DevOps?
What Is DevOps, Really?
Dave: Great question. You know these this is one that in our community we talk a lot about, especially for folks who are outside of quote-unquote “DevOps thought process”, knowing that it’s something that’s taking off as a force in the software world.
One of the things we do is to talk about how do we define DevOps. The biggest thing for me is DevOps means different things to different people and it’s all about context and perspective, where you come from and where you’ve been and what challenges you’re trying to solve. So when I meet somebody new who’s in this space and they’re starting to kind of either chant or evangelize to me without first getting a baseline perspective as to where I’m at and what I was doing and what I’m trying to solve, immediately has me question, “okay, are you trying to push your ideals down on me?”,
This is what DevOps means to me: getting folks to work together in an efficient collaborative manner to solve a common goal, period.
It has nothing to do with tools. It has nothing to do with process. It has nothing to do with frameworks. It’s all about getting people together, teaching context, having empathy, understanding what somebody’s doing, why they need to do it, and what what they’ve been doing in the past. You share your ways of doing it and then together when you have a sense of “okay, I know why this person has to do things, I know the reason why they’re thinking this way”, you can efficiently solve problems and for me that’s that’s what DevOps is to the core, right there.
Paul: So one one thing I heard from that is it starts with people, right? It doesn’t start with tools, it doesn’t start with how you’ve been doing it; it starts with people and really understanding the context and the perspective that they bring to the table. Is that right?
Dave: Yeah, Paul, you you nailed it right there. It starts, it continues, and it ends with people. Ultimately I take the concepts and the core principles of DevOps, and you can apply that to any industry, any product, any delivery, any manufacturing, and it really is bringing people together to work more efficiently to solve a common problem.
What Is DevOps Not?
Paul: And so actually people are doing that, you’ll hear the prevalence of these amalgam terms like DevSecOps, DevTestQAOps. And I kind of take issue with that in the sense that I understand how important terminology and clear labels for things. As a practitioner and engineer, as soon as somebody starts to blow out a term to mean “all the things”, my red flags get raised up instantly.
That doesn’t mean that [DevOps] doesn’t include other people, but can you tell us a little bit about how important the scope is of DevOps to you? And just kind of following that up with some context, I was able to speak to Ken Mugrage from the DevOps Days Seattle, and he was very clear about how if we blow it out into all the things, “DevOps” loses its value.
And so I put this to you: why is a pantheistic term, if DevOps grows to that, why is that a problem?
Dave: No, that’s a great thought. I want to take this back a little bit to identify why are all these actions added on, how and why this is how [DevOps] is being branded in this way. This was a discussion that I have, especially with growing teams.
One of the biggest things I talked about with organizations is, first and foremost, technically there is no DevOps engineers. So why label it that way?
There’s No Such Thing as a “DevOps Engineer”
When I started working with a lot more enterprises, I helped organizations transform their development to be much more modern so that they can have quicker release cycles and feedback. It’s one of the things that used to frustrate me, was like “hey, we need five DevOps engineers!”. That doesn’t mean anything to me, you got to explain on a day to day basis, what is this person doing, and ultimately, why are you labeling these folks as DevOps engineers?
And I I had some interesting feedback which came from the product marketing side. They were like, “Dave, we’re in the enterprise. We’re used to big long deploys of software in order to get it to our customers, and a lot of times we don’t know if our customers are even getting any value out of what we’re producing. When we’re releasing every year and waiting for six months to get the actual feedback from our customers, it doesn’t make any sense.”
So you see this large swath of folks trying to get into this space to build software quicker to have faster feedback to be able to add more value to end users.
These individuals don’t really understand this whole open source community, they don’t understand how the strength of the community is really the value.
“So we don’t know how to really market. We don’t know how to communicate to the group in a way for us to be able to blanket it all together. So we just scoped it into this thing and we call it #DevOps and everything gets that kind of label to it.”
From my experience what I’m starting to see is a lot more of these organizations who are specific to security, to testing, in a way of being able to catch and grasp that member of the audience, it’s “let’s throw it in, Dev and Sec Ops, Dev Quality Ops. What starts to happen in my mind and what I’m what I’m worried about is that people start to lose the real purpose.
Paul: So basically the exact same thing that happened to Agile. Everybody forgot to have agility as one of the core tenants that people check in on, on a regular basis such that they internalize that, and that is where their activities and their tools flow from, right?
Dave: Yes exactly. If you start to get too focused on the terminologies and the labeling of things and forget the context as to why you’re practicing it, ultimately the further down stream you get and the more generations that start to get folded into the process, they’ll start to lose the actual scope, “hey we’re trying to get people to to work together in a more collaborative manner to be efficient and to be able to deliver quickly.”
How to Be a Good DevOps (Citizen, Vendor, Employer)
Paul: Yeah, one thing that I did recently was put out an article (and thank you you, you had shared it to a number of people and I think that’s half the reason why I got some attention). It was essentially how to be a good DevOps vendor. It took the approach of looking at it from the customers perspective. The implementation of that was over a simplified customer journey and then chronologically through that journey, I went through and basically made statements from an outsider’s perspective onto different groups whether it be product, marketing, sales.
Back to your perspective, I get that it has to fundamentally start with people because people are what build teams and teams are what build software and software is what affects us. But the team affects us and individuals affect us, and so it does make sense to keep that as a core of value, to consider personal responsibility and also the responsibility of the team to have these cultural aspects present.
But unfortunately I think what happens is that we do need tools and you know, conferences are notorious for needing some kind of funding and becoming self-funding is really hard, and so out comes sponsor packages and I mean it’s an ecosystem. All software is eventually, in most people’s minds, going to make money and so this is where I was coming from, understanding that there is no such thing as a DevOps vendor or a DevOps tool or a DevOps job/position. Yet the fact is that when you’re closely aligned with the thinking of another person and “DevOps” is the term they’re using, it’s easy for these vendors to kind of bring that in and pull that into their messaging.
So I guess my my point of view on that is that we are gonna have to deal with that but it’s kind of a constant battle against the pantheism of trying to “all the things” a term [DevOps] but in the meantime we also do have to represent those tenants to more than just the developers and operations. If you really want to sell to developers and operations or teams that are looking, or they have internalized DevOps, they’re going to be looking at the world from this interesting perspective. And they’ll be looking across the tool chain to figure out who sounds like they’re blowing smoke up [you know where].
If a tool vendor or a service provider does not understand the core of DevOps, then their messaging, their selling process, their product ideation…it’s all not going to jive with the real market.
After after a recent Boston DevOps meetup we dove into this for what like an hour and a half, and just really talked about how do we actually do this. My concern is that when we start to move this into the enterprise (and by the way, the good principles of DevOps should be moveable to the enterprise, right? If they work, they work, and it’s a matter of fitting to context) that I think, while the core of it is culture, we can’t just live in this sort of kumbaya world.
We really have to figure out how to scale DevOps principals up and out into the enterprise setting so that, by the way, these good principles have a positive impact on things like automated insurance, things like machine learning in terms of healthcare, defense and government settings.
So I’m working on that on the side but in the meantime, what do you think about scaling to the enterprise? What does that even mean for DevOps?
How DevOps Is Re-writing Management Decisions
Dave: Yeah, that’s a great point. It’s an interesting challenge. There’s a lot of organizations who are facing it. Right now, I’m dealing with situations where we’re starting to see a lot of enterprise buy instead of trying to build it themselves. One thing they have is capital and resources. So the idea is, “if we don’t know or we can’t make it, it’s the bye versus build, like why go out and try to do what people already are being really successful in doing in something that we don’t understand too well? Let’s just go ahead and absorb some of these startups…”
Paul: Do you mean actually purchasing startups in order to just fill that technical gap in an organization? So I don’t want to name names, but I’m thinking of a very large enterprise that just recently bought up one of the most well-known API monitoring services out there, and people are freaking out like “oh gosh, what’s going to happen, are they going to de-culture this awesome group of guys and gals?”
Dave: I’m dealing with the same thing within an organization, a large security company buying a smaller more nimble security product with a lot of open source options. They’re putting out there trying to create groundswell to get this tool for free into the hands of engineers, let them play with it so they can understand how it works and create some kind of a swell within the engineering teams and then we’ll come up to the top start talking to the executives about, “Hey, what challenges are you facing in this broad space?”, where you’re trying to protect not only year your customers information but also information about your company.
As they start to have that type of dialogue, all of a sudden the executives within the organization starts to look down, talking to their engineering group and saying “hey, what do you know, what have you played with, what do you think is interesting, how do you think we should be solving this problem?” You’ve already created that initial lift of inertia in engineering, then they say hey let’s go with this product…we already know how it works, we’ve been you tooling around with it. Win-win, right?
So this is a completely different way of thinking of how enterprises used to be selling products into their customers. It was always a top-down approach…let’s talk to the executives who have the purchase power, float it down and then they’ll disseminate that information in the way that we roll it out into the engineering team. That’s how you could do it in the old-school way. Now in today’s new world, a lot of tools are available for you to play with for free and when enterprise organizations start to try to come into this space, they’re really kind of blindsided by this whole new content creation process.
Selling Into DevOps Takes Understanding DevOps
What I’m starting to see is they’re at least now recognizing we do not know how to sell to this to this community of this group. We know we really want to get into the space, we want to do it the right way, what do we do right and you know to your point with your article, I’ve shared your article with all of the enterprises that I’ve have been talking to me about this problem because I can’t teach them about the thought process of open source.
I mean, we can look back in the 60s, the MIT days, where the two groups kind of split off. A lot of us in the DevOps space already have the mentality of like “hey, you know we want to be able to share a lot of this stuff but we do want value for hard work we do”. But for the most part there’s different ways of doing it versus everything is being paid for with the enterprise mentality.
What I’m starting to experience is there’s a lot of organizations out there that are realizing it’s exponential value once they start to get into this community and…
the brand loyalty within the DevOps community is tremendous
…but the challenge that is in front of us right now is really the learnings piece and I’m thinking it’s a leadership issue (this is my own personal view). It’s enterprise leadership that needs to get out of the way and allow for new blood to come in to be able to understand the kind of movement. I’ve been doing a little, as much as I can to try to influence old leadership. It’s a challenge and a lot of it has to do with success syndrome. You’ve been doing it in certain way for decades. It’s a great case study that we’re gonna be able to kind of sit back and watch in the next five years
Calling All Researchers: Inclusion Means You Too!
Paul: Yeah and you know, there’s so much going on, no one person can do it alone. So without plugging any commercial products of any kind (that’s not my motion) I have started something called the iterativeresearch.org, which is essentially a bunch of contributors to research. As they go along, it could be lightweight contributions, simply just pocketing articles and getting into a feed of people who pay attention, it’s writers too, but the point is it’s not on a brand that’s connected to a pay-for services. And you know I would love, for this conversation to really start flowing in that direction because I think it takes many perspectives, right?
The core of this is it’s an inclusive conversation, not an exclusive one.
So understanding that you are a busy man and we’re at the top of our time, are there one or two things that you want to give a shout-out to or any particular resources that people can go to, events, communities, open-source forums, anything like that?
Get Out to a DevOps Tribe Near You!
Dave: Yeah, you know, thank you so much for the opportunity first and foremost, we’re gonna have to do it again! One of the things I really would highly recommend to folks who are interested in getting more involved, start to look at some local meetups that you have going on. There are some great folks within every community in whatever city, whatever small town, who are interested in sharing ideas and in thoughts in challenges. All you have to do is get out there and look. Go find your tribe! The biggest thing is don’t sit back and wait and sit on your hands and expect for interest to come to you.
The whole constant learner, the Kaizen mentality, be better tomorrow than you are today, be better today than you are yesterday. It lives and dies in DevOps and the way to do it is start to talk to folks who you’re not used to talking to.
Don’t be afraid get out there introduce yourself and have a good time. Life is learning.
Paul: Cool. So that’s David Frederick’s everyone and thank you David for spending the time with me. Do you prefer going by David, Dave?
Dave: Dave, David, either way.
Paul: Dave/David, I’ve really enjoyed it was great being able to spend some time. We’ll circle back. Thank you so much! Cheers!
Open source software (OSS) is a foundational part of the modern software delivery lifecycle. Enterprise teams with DevOps aspirations face unique challenges in compliance, security, reliability, and sustainability of OSS components. Organizations-in-transformation must have a complete picture of risk when integrating open source components.
This article explores how to continuously factor in community and ecosystem health into OSS risk analysis strategy.
The Acquisition Process for Open Source Software
Developers need to build on the successes and contributions of others. Having the flexibility to integrate new open source components and new versions of existing dependencies enables teams to go fast, but external code must be checked and validated before becoming part of the trusted stack.
Including someone else’s software is an important moment of engagement. Enterprises typically wrap a formal ‘acquisition’ process around this process, where the ‘supplier’ (the entity who provides the software/service) and the ‘acquirer’ (the entity who wants to integrate the software/service) contractualize.
Though there are already commercial approaches to introducing software packages safely by companies like Sonatype,Black Duck, and others, my question extends beyond the tools conversation to encompass the longer-term picture of identifying and managing risk in software delivery.
Enterprises care deeply about risk. Without addressing this concern, engineering teams will never actualize the benefits of DevOps.
This is a tangible application of the need for DevOps to not only apply at an individual team level, but in the broader organization as well. It takes alignment between a team who needs software and teams providing compliance and legal services to all do so in an expedient manner that matches the clock speed of software delivery.
Communities Empower Enterprises to Address this Gap
Today in a Global Open Source Governance Group Chat, I asked the question:
“What are some methods for determining how significant a supplier/vendor OSS and community contributions are, relative to acquirer confidence?”
This question stems from my involvement with the IEEE 2675 working group, particularly because I see:
Prolific use of OSS use in DevOps and in enterprise contexts
Reluctance and concern (rightly so) around integration of OSS in enterprise software development and operation in production
The convergence of compliance and automation considerations
How important transparency and collaboration is to the health of OSS, but also to the supply and acquisition processes in a DevOps lifecycle
As open source projects (like Swagger/OADF for instance) become increasingly important to enterprise software delivery, health and ecosystem tracking also becomes equally important to any new components being introduced.
My point-of-view is that organizations should prepare a checklist for software teams to construct a complete picture of risk introduced by OSS (not to mention proprietary) components. This checklist must include not only static analysis metrics but support, engagement, funding, and contribution considerations.
Measuring OSS Project + Community Health
The group had many suggestions that I wouldn’t have otherwise thought about, another reason for more people getting involved in dialogs like this.
There are already providers of aggregate information on open source community health and contribution metrics such as CHAOSS, a Linux Foundation project, and Bitergia. This data can be integrated easily into dependency management scripts in Groovy, npm, Ant, Maven, etc. and at the very least written in to a delivery pipeline as part of pre-build validation (BVT is too late).
The group also identified some key characteristics of OSS community health not necessarily tracked by established services, such as:
Same day response on reported issues, even if it’s simply acknowledgement
PRs under the “magic number” of 400 lines of code…tends to be the limit for # of bugs and useful feedback
Outage response, sandbox availability
Distribution of component versions across multiple central repositories
More to Come…From YOU
As I integrate both my own learnings and other voices from the community into the larger Enterprise DevOps conversation, the one thing that will be missed is YOUR THOUGHTS, whether your in a large organization or simply in a small team.