Folding Open Source into Enterprise DevOps

Open source software (OSS) is a foundational part of the modern software delivery lifecycle. Enterprise teams with DevOps aspirations face unique challenges in compliance, security, reliability, and sustainability of OSS components. Organizations-in-transformation must have a complete picture of risk when integrating open source components.

This article explores how to continuously factor in community and ecosystem health into OSS risk analysis strategy.

The Acquisition Process for Open Source Software

Developers need to build on the successes and contributions of others. Having the flexibility to integrate new open source components and new versions of existing dependencies enables teams to go fast, but external code must be checked and validated before becoming part of the trusted stack.

Including someone else’s software is an important moment of engagement. Enterprises typically wrap a formal ‘acquisition’ process around this process, where the ‘supplier’ (the entity who provides the software/service) and the ‘acquirer’ (the entity who wants to integrate the software/service) contractualize.

Though there are already commercial approaches to introducing software packages safely by companies like Sonatype, Black Duck, and others, my question extends beyond the tools conversation to encompass the longer-term picture of identifying and managing risk in software delivery.

Enterprises care deeply about risk. Without addressing this concern, engineering teams will never actualize the benefits of DevOps.

This is a tangible application of the need for DevOps to not only apply at an individual team level, but in the broader organization as well. It takes alignment between a team who needs software and teams providing compliance and legal services to all do so in an expedient manner that matches the clock speed of software delivery.

Communities Empower Enterprises to Address this Gap

Today in a Global Open Source Governance Group Chat, I asked the question:

“What are some methods for determining how significant a supplier/vendor OSS and community contributions are, relative to acquirer confidence?”

This question stems from my involvement with the IEEE 2675 working group, particularly because I see:

  • Prolific use of OSS use in DevOps and in enterprise contexts
  • Reluctance and concern (rightly so) around integration of OSS in enterprise software development and operation in production
  • The convergence of compliance and automation considerations
  • How important transparency and collaboration is to the health of OSS, but also to the supply and acquisition processes in a DevOps lifecycle

The group, expertly facilitated by Lauri Apple, also included key insights from Paul Burt and Jamie Allen. A log of the conversation can be found on Twitter.

As open source projects (like Swagger/OADF for instance) become increasingly important to enterprise software delivery, health and ecosystem tracking also becomes equally important to any new components being introduced.

My point-of-view is that organizations should prepare a checklist for software teams to construct a complete picture of risk introduced by OSS (not to mention proprietary) components. This checklist must include not only static analysis metrics but support, engagement, funding, and contribution considerations.

Measuring OSS Project + Community Health

The group had many suggestions that I wouldn’t have otherwise thought about, another reason for more people getting involved in dialogs like this.

There are already providers of aggregate information on open source community health and contribution metrics such as CHAOSS, a Linux Foundation project, and Bitergia. This data can be integrated easily into dependency management scripts in Groovy, npm, Ant, Maven, etc. and at the very least written in to a delivery pipeline as part of pre-build validation (BVT is too late).

And there is honest, hard-hitting research on open source software…which you should take the time to read….from Nadia Eghbal under the Ford Foundation in a report called ‘Roads and Bridges: The Unseen Labor Behind Our Digital Infrastructure‘. If you don’t have time to read it, buy some text-to-speech software and listen to it when you’re in transit.

The group also identified some key characteristics of OSS community health not necessarily tracked by established services, such as:

  • Same day response on reported issues, even if it’s simply acknowledgement
  • PRs under the “magic number” of 400 lines of code…tends to be the limit for # of bugs and useful feedback
  • Outage response, sandbox availability
  • Distribution of component versions across multiple central repositories

More to Come…From YOU

As I integrate both my own learnings and other voices from the community into the larger Enterprise DevOps conversation, the one thing that will be missed is YOUR THOUGHTS, whether your in a large organization or simply in a small team.

Please share your thoughts in the comments section below! Or ping me up on Twitter@paulsbruce or LinkedIn.

More reading:

AnDevCon: Espresso Is Awesome, But Where Are It’s Edges?

For my presentation at AnDevCon SF 2016, I focused on how Espresso represents a fundamental change in how approach the process of shipping software that provably works on a mobile ecosystem that is constantly changing.

The feedback was overwhelmingly good, many people who stopped by the Perfecto booth before or after our talk came to me to discuss topics I raised. In other words, it did what I wanted, which was to provide value and strike up conversation about how to improve the Android UI testing process.

If you’re pressed for time, my slides can be found below or at:
bit.ly/espresso-edges-andevcon

Gluecon 2016: Melody Meckfessel on Serverless Technology

I had the opportunity to attend Melody Meckfessel’s presentation at Gluecon 2016 last week. As someone with years of experience at Google, when she speak, smart people listen.

[paraphrased from notes]

We should all be optimistic about open source and serverless technologies. Kubernetes, an example of how to evolve your service, a means to an end, housing processes internally, making management easier for developers to deploy and scale horizontally, to update gracefully without causing downtime. One of the themes in the future of cloud-based computing is that is increasingly open source which makes it easier for developers to influence and contribute the software stack as it’s evolving. Our focus is switching to managing applications and not machines.

“…the future of cloud-based computing…is increasingly open source which makes it easier for developers to influence and contribute the software stack as it’s evolving.”


(photo credit: @rockchick322004)

Serverless means caring about the code, embedded in all cloud platforms. In App Engine, you run code snippets and only get charged for your app or service as it scales up or down. With container engine, you run containers but don’t specify the machine it runs on. In the future, we’re not going to be thinking or talking about VMs. What does it mean for us developers? It will enable us to compose and create our software dynamically. As one of the most creative and competitive markets, software puts us in the same room and presents us with the same challenge: how to make things better, faster.

Melody tells a story about a project she was working on where they were using a new piece of hardware. They were having trouble scaling during peak load, holiday traffic being really important to their particular customer. They were constantly scrambling and for the first 6-9 months, she spent her time on primarily DevOps related work. This was frustrating because she wanted to work on features, seeing the potential of what she could do, and she could never quite get to that.

As developers, we are constantly raising the bar on the tools we use to release, and correspondingly our users’ expectations increase. Also as developers, we have the ability to create magic for users, predicting what they want, putting things in context, and launching the next great engaging thing. In a serverless future, developers will be focused on code, which means that PaaS (platform as a service) will have to evolve to integrate the components it’s not doing today. To do that, developers will need to shift Ops work from themselves more to cloud providers, being able to pick and choose different providers. Developers come to a platform with a specific language that they’re productive in, so this is a place where PaaS providers will have to support a variety of runtimes and interpreters. App Engine is the hint and glimmer of creating an entirely new developer experience.

“In a serverless future, developers will be focused on code…will need to shift Ops work from themselves more to cloud providers.”

What does a serverless future mean for the developer experience?

We will spend more of our time coding and have more and more choice. Right now, we’re in a world where we still have to deal with hardware, but as that changes, we’ll spend more of our time building value in our products and services and leave the infrastructure costs to providers and the business. If we want to do that, developers need to make it very easy to integrate with machine learning frameworks and provide analytics. Again, to do this, we need to free up our time: hence NoOps. If we’re spending all out time on ops, we’re not going to get to a world where we can customize and tailor our applications for users in the ways they want and expect. This is why NoOps is going to continue to go mainstream.

If we’re spending all out time on ops, we’re not going to get to a world where we can customize and tailor our applications for users in the ways they want and expect.

Think about if you’re a startup. If you’re writing code snippets, you may not need to think about storage. You don’t need to worry about security because the cloud providers will take care of that for you. Some wifi and laptops, and that’s your infrastructure. You’ll have machine-oriented frameworks to allow that app to be really amazing. This idea that you can go from prototype to production-ready to then scale it to the whole planet. Examples of really disruptive technology because they were enabled to do so by cloud resources.

To get there, we’re going to have to automate away the configuration and management and the overhead. We still have to do this automation. We have work to do there.

Multi-cloud in NoOps is a reality.

People are using mixed and multiple levels of on-prem IT and hybrid cloud; the problem is that the operational tools are siloed in this model though. This makes it very hard to operate these services. As developers, we should expect unifying management…and we should get it. Kubernetes is an example, just another way for teams to deploy faster. Interestingly, we’re seeing enterprises use Docker and Kubernetes on-prem; the effect of this is that it will make their apps and services cloud-ready. When they are faced with having to migrate to the cloud, it will be easy to do it.

“…we’re seeing enterprises use Docker and Kubernetes on-prem…it will make their apps and services cloud-ready”

Once you’re deployed like this though across multiple clouds and service providers, you’ll need to have an easy way to collect logs and perform analysis; StackDriver is an example of this. It’s this single-pane-of-glass view that enables developers to find and fix issues fast, to see what’s happening with workloads, and to monitor more effectively. Debugging and diagnosing isn’t going to go away, but with better services like Spinnaker and App Engine, we’ll be able to manage it more effectively.

Spinnaker providers continuous delivery functionality, tractability throughout our deployment process, it’s open source. As a collaboration with Netflix, Melody’s team worked hard on this. It was a case of being more open about the software stack we’re using and multiple companies coming together. We don’t all need to go build our own thing, especially when we share so many common problems.

Debugging and diagnosis

The problem is that there’s all this information across all these sources and it’s hard to make sense of it. It’s usually in a time-critical situation and we have to push out a quick fix as soon as we can. Part of streamlining that developer and debugging experience in the future cloud is having the tools at our disposal. These are things like being able to trace through a complex application for a performance issue, going back through the last 3 releases and see where the slow-down started, or production debuggers where you can inspect your service that’s running in the cloud and go to any point in the code to look through variable and see what’s happening. Error reporting as well needs to be easier, as errors can be spread across multiple logs, it’s hard to see the impact of the errors you’re seeing. Errors aren’t going away, so we need to be able to handle them effectively. We want to reduce the effort it takes to resolve these errors, we want to speed up the iteration cycles for finding and fixing the errors, and provide better system transparency.

“The faster we can bring the insight from these motions back in to our source code and hence optimize, those are all benefits that we are passing through to users, making us all happier.”

In the NoOps, serverless future, not everything will be open source, but we will expect that 3rd party providers should work well with all the cloud platforms. Meledy’s team is currently working on an open source version of their build system called Basil. Right after their work on Spinnaker in collaboration with Netflix, they’re looking for other opportunities to work with others on open source tools.

What does the world look like 5-10 years from now?

We’re not talking about VMs anymore. We’re benefiting from accelerated hardware development. We’re seeing integration of data into apps and compute so that we can have better insight into how to make applications better. It’s easier for machine learning to be embedded in apps so that developers can create that magical software that users expect. We don’t have to talk about DevOps anymore. We will have more time to innovate. In an open cloud, you have more opportunity to influence and contribute to how things evolve. Like thinking back on how there used to be pay phones and there aren’t anymore, the future of development is wide open, no one knows what it will look like. But contributions towards the future cloud are a big part of that future and everyone’s invited.

More resources:

Minimum Viable Open Source

It was about 7 months ago I started to feel myself drawing a line about how I use the words “open source”. 7 months. Certainly in my copy, my writing, and my ideas for future projects (not just programming projects), these words infer meaning beyond what most people think about when they use them.

Originally, open source software was just a bunch of scientists sharing useful stuff. Now its a method of brand extension. How far we’ve all come. It used to mean the freedom to do what one thought was worth doing. Now it seems more synonymous with another word: shackles.

So I want to put it down here, a point in time, where people can see my progression. To call something open source, I expect that:

  • it’s IP/source is available freely without signup
  • it is moderated by the community of its own contributors, not one entity
  • it is maintained separate from other commercial product/service lifecycles
  • does not share the identity of its benefactors  consent
  • does not contain EULAs that non-technical and non-legal persons can understand

I have no problem with companies who want to use “open source” to their brand advantage, but to not respect these few core principals is to be ‘half in’ and very transparent in its intent.

Other reading:

The Cost of Not Changing Things Up

This week, I realized that for years, I’ve been looking at life changes in terms of cost. How much will it cost to move, how much will it cost if I quit, how much will it cost if I fail.

Nothing of worth comes on a silver spoon

Despite warnings and signs, friends and teachers, and despite being part of moments in spacetime I would gladly revisit, my aha moments always necessarily originate internally, most often when I put myself in challenging situations. I always want to know what’s in the pita.

While waiting for an officer to write me a bogus citation today, it hit me that much as in the same way I measure things from both sunk vs. opportunity cost perspectives in business, I forgot to do that with my home and personal life. The cost of what I’m doing in one place is more than what it costs me emotionally, physically, and mentally; it’s costing me true happiness that I’m leaving on the table.

Forget silver, there is no spoon

I love this thing. It’s not my family (of course I love them), it’s literally a thing. I have fallen in love with an open source thing. That passion has caused me to ignore the bigger picture. And who the fuck loves a thing over themselves?

When someone or something is tied to a prior transformation, it’s hard to give it up. It’s an edifice, a token. It’s also just a thing. And since this particular thing is open source, it’s free (for now at least) so at least there’s no sunk cost for me to keep using it.

There may be no spoon, how about a knife

I feel like a snake during ecdysis, or maybe a pile of phoenix ash. Either way, change is coming, and bitches, watch what happens between here and Denver. I have something no one is expecting. I will cut through so much wrapping paper that the present inside will be a fun little aftershow.

Semi-related reading:

[Talk] API Strategy: The Next Generation

I took the mic at APIStrat Austin 2015 last week.

A few weeks back, Kin Lane (sup) emailed and asked if I could fill in a spot, talk about something that was not all corporate slides. After being declined two weeks before that and practically interrogating Mark Boyd when he graciously called me to tell me that my talk wasn’t accepted, I was like “haal no!” (in my head) as I wrote back “haal yes” because duh.

I don’t really know if it was apparent during, but I didn’t practice. Last year at APIStrat Chicago, I practiced my 15 minute talk for about three weeks before. At APIdays Mediterranea in May I used a fallback notebook and someone tweeted that using notes is bullshit. Touché, though some of us keep our instincts in check with self-deprecation and self-doubt. Point taken: don’t open your mouth unless you know something deep enough where you absolutely must share it.

I don’t use notes anymore. I live what I talk about. I talk about what I live. APIs.

I live with two crazy people and a superhuman. It’s kind of weird. My children are young and creative, my wife and I do whatever we can to feed them. So when some asshole single developer tries to tell me that they know more about how to build something amazing with their bare hands, I’m like “psh, please, do have kids?” (again, in my head).

Children are literally the only way our race carries on. You want to tell me how to carry on about APIs, let me see how much brain-power for API design nuance you have left after a toddler carries on in your left ear for over an hour.

My life is basically APIs + Kids + Philanthropy + Sleep.

That’s where my talk at APIstrat came from. Me. For those who don’t follow, imagine that you’ve committed to a long-term project for how to make everyone’s life a little easier by contributing good people to the world, people with hearts and minds at least slightly better than your own. Hi.

It was a testing and monitoring track, so for people coming to see bullet lists of the latest ways to ignore important characteristics and system behaviors that only come from working closely with a distributed system, it may have been disappointing. But based on the number of conversation afterwards, I don’t think that’s what happened for most of the audience. My message was:

Metrics <= implementation <= design <= team <= people

If you don’t get people right, you’re doomed to deal with overly complicated metrics from dysfunctional systems born of hasty design by scattered teams of ineffective people.

My one piece of advice: consider that each person you work with when designing things was also once a child, and like you, has developed their own form of learning. Learn from them, and they will learn from you.

 

HackCU : An Example of Student Leadership

In an unused downstairs side room of a hotel, I listened to students from the University of Colorado express their desire to change the world, and their concerns about commercial interests moving in to poach talent from hackathons.

Alex Campbell, Alex Walling, and Nika Shafranov.

[Download as MP3]

HackCU is a Boulder-based hackathon incubator program that seeks to educate, inspire, and connect local students to technology and hands-on skills. It is at the cutting edge of technical education. It is student-led. It is well intentioned.

It is a way for students to arm themselves in the digital workforce when they can’t trust the technical market to treat them properly as employees. They teach each other how to code, design, and prove the value of their own technology.

I think this is great. These are the right people to be running this sort of thing. I’ll be keeping my eyes open about hackathon politics in the local area thanks to my conversations with these fine engineers.

My goal in engaging the two Alex’s and Nika, with help from Maria Sallis of StartupDigestCO and Lorinda Brandon‘s wise words, was to assist these students in any way a storyteller like me can: network the right people together, understand their challenges, and help them tell that story to a wider audience.

Much like the open source community (which I am far less a part of than I’d like to be), the communities of student-led hackathons are a brilliant place to hang out and listen, intellectually stimulating and ethically challenging. My own exploration of this space will take time.

Advocation for Open Source

Getting people to understand the true value of open source goes beyond just making easily digestible bullet points. I don’t mean the hyper-commercialized version of open source that dominates the software market today. I mean that open source software is a way in to the mind and heart of a programmer.

Programmer != “developer”

Programmers love code. They love to fix things. They love when something can be expressed in concrete terms. They love other people, though not all of them know that they express this through their code. They hate when something doesn’t work properly, just like a true engineer does. They dislike bullshit, in any form. They hate when something that should be free, isn’t.

DEF CON 22 pictures - AST_3546

The term “Developers” on the other hand is a concept that certain people invented, constructed, to create a marketplace of people that do first instead of consider first. Developers, not code, have been the new tech market for many years. Even well-meaning technical people sadly fall prey to the wholesale commoditization of humans who speak in computer languages and are easily caught up in the dark ocean of enterprise software.

I wish all the developers in the world well as I sail away and find a more honest place to keep my self and my code.

This guy worked as a developer, then he quit work

I have contributed to many open source projects over the years. Some I have taken too personally, some I have ignored, and some I squandered. Sadly, for me and this post, it cannot be verified, as I have used many faux-names and anonymous accounts to do so. I can’t even remember or trace them even if I tried. [heartache] [pause] [sigh]

In my youth, I didn’t realize how important contribution was. I thought that the concept of the ‘self’ was something to be avoided, ignored. Since that time, I understand how wasteful that viewpoint was. Contribution is a requisite now.

Open source software is the language of people who want to work together.

The problem with money

There’s something inherently wrong with a company that doesn’t regularly contribute back to the community, something very wrong and very rotten. Money is a really effective motivator. It is often also used as the only indicator of value in a marketplace. Shame on us for not looking deeper, because there are many metrics to human behavior and interest, the last of which is where people put their money. Ask anyone in sales, they’ll tell you that people’s interest is ultimately, but not initially gauged by their financial commitment. People will tell you if your thing, whatever that is, is bad or worth investing in; you just have to listen to the signs. And there’s no better way to do that than open source contributions as a metric for the veracity of your thing in the market..

I advocate for open source

One theme that emerges from my years into software is that contribution is key. Commercial, open source, whatever. Teams are made up of people, and to do something great, the more engaged and honest people you find, the better off you’ll be.

Side note: It is not enough to treat a non-profit as your contribution. The maintstream NPO mentality is that money is only necessary when it’s needed, desperately, in salaries, in budgets, and in planning. People are not created equal. We start at different places, with different perspectives, and claw our way to an egalitarian place. Some people have to work harder than others.

It is the job of those who don’t work as hard to help out those who work too much.

That is the spirit of open source to me. To help. That’s why I advocate open source now. It is help that causes people to put there money where their mouth is, in your product. To make money is to help first; the software equivalent of that is to contribute to open source, whatever way you can.