7 minute read

Probably not unlike you, every day I work with folks caught in a clash between organizational processes and technology imperatives. "We have to get this new software up and running, but the #DevOps group won't give me the time of day."

Large organizations don't have the luxury of 'move fast, break stuff'; if they did, their infrastructure, security, financial, and software release processes would be a chaotic mess...far more than usual. But how does one 'move fast' without breaking enterprise processes, particularly ones that they don't understand?

Enterprise, Know Thyself

The answer is simple: encourage engineers to always be curious to know more about their environment, constraints, and organizational culture. The more you know, the more nimble you'll be when planning and responding to unanticipated situations.

Today I had a call with a health care company, working to get docker installed on a RHEL server provisioned by an infra team. What was missing was that the operator didn't know that the security team using Centrify to manage permissions on that box required tickets to be created to grant 'dzdo su' access for a very narrow window of time. Additionally, the usual 'person to connect with' was off on holiday break, so we were at the mercy of a semi-automated process for handling these tickets, and because they had already put in a similar request in the past 7 days, all new tickets would have to go through a manual verification process. This frustrated our friend.

The frustration manifested in the form of the following statement:

Why can't they just let me have admin access to this non-production machine for more like 72 hours? Why only 2 meaasly hours at a time?

- Engineer at an F100 health care organization

My empathy and encouragement to them was to "expect delays at first, don't expect everyone to know exactly how processes work until they've gone through them a few times, but don't accept things like this as discouragements to your primary objective."

If everything were easy and no problems existed, kind words might be useless. When things are not working that way, knowing how to fix or overcome them goes a long way, just like a kind word at the right time. We crafted an email to the security team together explaining exactly what was needed AND WHY, as well as an indication of the authority and best/worst case timelines that we were operating under, and a sincere thank you.

Enterprise "DevOps" Patterns that Feel Like Anti-Patterns

In my current work, I experience a lot of different enterprise dynamics at many organizations around the world. The same themes, of course, come up often. A few dynamics I've seen in play when enterprises try to put new technology work in a pretty box (i.e. consolidate "DevOps engineers" into a centralized team) are:

  1. Enterprise DevOps/CloudOps/infra teams adopt the pattern of "planned work", just like development teams, using sprints and work tracking to provide manageable throughput and consistency of support to other organizational 'consumers'. This inherits other patterns like prioritization of work items, delivery dates, estimable progress, etc.
  2. Low/no context requests into these teams get rejected because it's slow/impossible to prioritize and plan based on ambiguous work requirements
  3. The amount of control and responsibility these teams have over security and infrastructure systems the organization is often considered "high risk", so they're subject to additional scrutiny come audit time

That last point about auditing, particularly the psychological impacts on 'move fast' engineers, cannot be understated. When someone asks you to break protocol 'just this one time', it's you that's on the hook for explaining why you took action to do so, rarely the product owner or director who pressured the engineer to do it.

Technical auditors that are worth anything more than spit will focus on processes instead of narrow activities because to comb through individual log entries is not scalable...but verifying that critical risk mitigative processes are in place and checking for examples of when the process is AND isn't being followed...that's far more doable in the few precious weeks that auditing firms are contracted to complete their work.

The More You Know, The Faster You Can Go (Safely)

An example of how understanding your enterprise organization's culture improves the speed of your work comes from an email today between two colleagues at F100+:

Can you confirm tentative dates when you are planning to conduct this test? Also will it take time to open firewall, post freeze incident tickets can be fast tracked?

- Performance Engineering at Major Retailer

This is a simple example of proper planning. Notice that the first as is for concrete dates, an inference that others also need to have their shit together (in this particular case because they're conducting a 100k synthetic user test against some system, not a trivial thing in the slightest). The knowledge that firewall rules have to be requested ahead of time, and to notify incident response that potential issues reported may be due to the simulation, not real production traffic, comes from having experienced these things before. Understanding takes time.

Another software engineer friend of mine in the open-source space and I were discussing the Centrify thing today, and he asked: "why can't they just set up and configure this server with temporary admin rights off to the side, then route appropriate ports and stuff to it once it's working?" Many practitioners in the bowels of enterprises will recognize a few wild assumptions there, and in no way is this a slight of my friend, but rather an example of how different thinking is from two very different engineering cultures. More specifically, those who are used to being constrained as opposed to those who aren't often have a harder time collaborating with each other because they're reasoning is predicated on very different past experiences. I see this one a lot.

DevOps Is an Approach to Engineering Culture, not a Team

This is my perspective after only 5yrs of working out what "DevOps" means. I encourage everyone to find their own by having their own journey of curiosity, keyboard work, and many conversations.

There is and never should be a DevOps 'manifesto'. As Andrew Clay Shafer (@littleidea) once said, DevOps is about 'optimizing for people', not process or policy or one type of team only. Instead of manifesto bullet points, there are some clear and common principles that have stayed the test of time since 2008:

  • A flow of work, as one way as possible
  • Observability and Transparency
  • Effective communication and collaboration
  • A high degree of automation
  • Feedback and experimentation for learning and mastery

Some of the principles above come from early work like The Phoenix Project, The Goal, and Continuous Delivery; others come from more formalized research such as ISO and IEEE working groups on DevOps that I've been a part of over the past 3 years.

I don't tend to bring the "DevOps is not a team" bit up when talking with F100s primarily because:

  • it's not terribly relevant to our immediate work and deliverables
  • enterprises that think in terms of cost centers always make up departments, because "we have to know who's budget to pay them from and who manages them"
  • Now that DevOps is in vogue with various IT leaders and just like the manifestation of Agile everywhere now, DevOps is perceived as 'yet another demand to do things differently from management', so after being restructured, engineers often have enough open wounds that I don't need to throw salt on
  • if this is how people grok DevOps in their organization, there's little I as an 'outside' actor can do to change it...except maybe a little side-conversation over beers here and there, which I try to do as much as appropriately possible with receptive folks

However, as an approach to engineering culture, DevOps expects people to work together, to "row in the same direction", and to learn at every opportunity. As I stated at the beginning of this post, learning more about the people and processes around you, the constraints and interactions behind the behaviors we see, being curious, and having empathy...these things all still work in an enterprise context.

As the Buddha taught, the Middle Path gives vision, gives knowledge, and leads to calm, to insight, to enlightenment. There is always a 'middle way', and IMO is often the easiest path between extremes to get to the place where you want to be.

You may also enjoy