On Volunteering for Tech Community Work

DevOpsDays Boston volunteers

This is my attempt to distill what I’ve learned over the past two years of contributing to maintaining and improving the local DevOps community in Boston. As in all cases, what we think we “know” at a point in time can/may/should change as our journey progresses. Rather than prescriptive, this article is descriptive of my current approach.

TL;DR

  • do it for ‘right’ reasons that align with community values
  • really know what you’re getting into and commit proportionally
  • be prepared to spend time and energy, particularly on personal adaptation
  • manage your life work-in-process limit carefully
  • expect something unexpected out of it

A True Community is Tangible

There are many definitions of “community”, but here’s mine at this point:

A group of people dedicated to sharing and improving along with each other.

– me, provisionally as of 2019

If helping to do this sounds easy, it is not. Dedication means graciously and humbly collaborating with other organizers, diligently following up with a million little details, listening and understanding where people in the community are coming from, actively seeking personal improvement and synthesizing into volunteer work, responding quickly to important issues in the community and of course, making/taking time from your personal and professional life to do these things reliably.

For an organizer, these things are very tangible because they cost us what we can never get back: time. For other community members, the value comes from elements like knowledge sharing, conversations, job opportunities, or even a sense of simply being part of a community. Showing up at a meetup event or a casual coffee chat is the first step, and understanding why you enjoyed brings you back.

One of the most concrete, tangible outcomes of being part of the community I work in is that there’s always an opportunity to learn something new and useful, which by definition you can’t predict. The more you give, the more you get. The more you’re open-minded, the more mindful and receptive to good ideas you become. The more you listen, the more you hear.

Fostering Unity and Diversity at the Same Time

We all come to the table with a variety of backgrounds, experiences, and perspectives. This is the true strength of a healthy community. Balancing time constraints and the distribution curve of appetites for themes is tricky, but doable when facilitators are dedicated and have the interests of each other (including the whole community) in mind.

There’s usually a unifying set of core topics and themes related to some common interests (such as learning new tech or how to deal with office culture or professional improvements). Straying too far outside this focus often contributes to our “variety” getting in the way of sharing and improving, though sometimes including topics seemingly unrelated to the tech side of the conversation can provide opportunities for divergent thinking and new knowledge.

An Organizer’s Roles and Responsibilities in the Community

At a small scale, this may happen organically, but as the group scales, this doesn’t happen without organizers. A community organizer is someone who:

  • looks out for the best interests of all members of the community
  • is dedicated to creating more value than they capture [1]
  • ensures that underrepresented perspectives have a voice
  • deals with the logistics of shared spaces (events, chats)
  • ensures that the values and principles in the Code of Conduct are respected
  • develops a plan of succession and redundancy in organizing responsibilities
  • insulates the community from negative or unwanted solicitations

I’ve had to great privilege to work with a bunch of other organizers and volunteers, and one of the key things I see make up for the aggregation of individual intents and sometimes dysfunctions is the willingness to come back and do/be better.

One thing that’s always a challenge is to maintain a pool of organizers greater than one or two active individuals. Think of it as resiliency engineering, a topic close to my personal wheelhouse. The more trustworthy and reliable individuals are involved, the more trustworthy and reliable the outcome.

Thrash vs. Progress

Everyone brings their own personal interests into a group equation. When the guiding principles include alignment and responsibility, it works really well for everyone. When someone puts their personal interests above the common good, all is not lost, but causes a lot of “thrash”. I define thrash as work which ultimately does not progress and improve the common good.

Some folks need to thrash around with new ideas and values, and this is okay too. Healthy synthesis of new ideas and values is important. Provided that there’s room for people to bring these ideas to the table and that progress on existing values is in play, we all win.

Getting Ahead of Toxicisity

I’ve worked in a number of places and communities where behavior is sub-par. When something is introduced which causes a body (or community) to make a backward motion towards it’s shared and the common good, particularly on a repeated basis, I would call this “toxic”. In a DevOps mindset, this can happen too, where anti-patterns such as burnout and distrust crop up because little details aren’t being addressed.

In DevOps communities, many are familiar with the Westrum model of organizational culture: pathological, bureaucratic, and generative. Though often misused to label groups and people at an aggregate level, the most useful employment of this lens IMO is around behaviors and activities. Those can be changed, those can be provided guidance where people have specific decisions to make.

Two examples from our local meetup come to mind: use of alcohol and meritocratic rhetoric. Though innocuous in small amounts, both quickly detract from community spirit and sense of safety in our culture.

In one thread, someone suggested that we bring our own bottles of scotch. I love my happy hours the same as warm-blood, but immediately before that, I witnessed and heard from a number of other community members that an unannounced and unplanned open bar at a meetup detracted from fostering a safe space. I quickly hopped on this, making sure that we all consider how this idea contributes to values in our Code of Conduct. Call me the culture cop, but after collaborating with other organizers on the approach, it was clear that we wanted to get ahead of where it was going and pivot the conversation back to the purpose of the community.

In another backchannel thread with community members, we discussed the rising trend of conversations assuming that everyone had the same opportunities leading to a notion that all people would be rewarded based on the merit of their works. In the current tech culture, meritocracy is rampant, most notably due to open-source mainstays such as the Linux Code of Conduct and revulsion of GNU founder R.M.S after a career of misogynistic and belligerent behavior. Not everyone can just walk up to a table and be valued by those there. As a common good of our community, we do our best to steer away from the ‘meritocracy lens’ and bring conversations back to a place that is useful for all members of the community.

Improvement Means Earning and Granting Trust

One of my biggest goals is to find someone better than me at these things that not only fulfills above bulleted organizer requirements but that I can trust to improve and grow the community in ways I can’t. Knowing that I probably won’t be able to keep up the same level of commitment to our community indefinitely, succeeding myself with other trustworthy individuals is not a zero-effort endeavor in the slightest. Nothing happens overnight, and the best time to do something truly good is always ‘now’.

It’s definitely been interesting to co-organize the meetup, facilitating events and threads, hopefully earning an amount of trust in our community about why I’m spending my time and energies on it. What I’ve learned is that granting someone trust is a very different proposition than earning it. One thing suggested to me in a backchannel was that “ally is a term someone uses on you, not something you “. After bouncing this around in the back of my mind for months, I’ve realized that I don’t need to focus on wait for people to ascribe community members (especially organizers) trust, we just need to do the best we can and demonstrate trustworthy behavior.

As one of 6 organizers of the yearly DevOpsDays Boston event, I was in a place to understand that forging an initial relationship with the Inclusive Tech Lab and Resilient Coders communities was a great direction (most props go to Laura Stone, another co-organizer of the meetup and event). The heart and spirit of these other groups align with the values and demonstrated levels of commitment that I know are required to drive the community forward.

Listening to and Integrating Feedback

Imagine all the times that someone spoke words that you didn’t understand…thing you weren’t in a place to receive. What a shame. What do you do about that?

Mostly always looking to get better, but one thing I’ve realized is that the moments my mouth is open should be marginal compared to time I spend hearing what others have to share. If doing it right, the times I speak should be guided by three questions:

  • does this have to be said (and why)?
  • does this have to be said by me?
  • does this have to be said right now?

Though everyone suffers from subjectivity (implicit in the above bullets), listening and empathizing with what you know about someone’s perspective increases the “you AND them” funnel for new and important ideas. Also, reading or listening to lots of books and blogs also widens perspective.

Get Involved, Be Good

Though not a requirement to attending, I’m surprised at the number of regular attendees who haven’t actually read the Code of Conduct but still naturally adhere to the values and principles within. I take this as a sign of common interest in the community but don’t take for granted the responsibility to make sure behaviors are inline.

If you’re thinking of helping, definitely reach out. If not, still, leave comments. I appreciate feedback and engagement on approach to facilitating in the community, always.

This is Why #DevOps

Since well before 2008, DevOps as a keyword has been growing steadily in mindshare. This post is my version of a landing page for conversations I have with folks who either know little about it, know a lot about it, or simply want to learn why DevOps is not a hashtag buzzword.

TL;DR: if you’re not interested in reading a few paragraphs of text, then nevermind. If you really must jump to my own articulation of DevOps, click here.

DevOps Is a Question, Not an Answer

The word “DevOps” is a starting point in my conversations with folks along the journey. Whether it’s over obligatory beers with local community members after a monthly meetup, in a high-pressure F100 conference room, in my weekly meetings with the IEEE, or internally in organizations I work with and accept benefits from, “DevOps” is often the start of a really important dialog. It’s a question more than a statement, at least the way I use it day to day, spiked with a small challenge.

It’s a question of “what are you doing now, and where do you want to be”. It’s a question of “how are the things you’re doing now incrementally improving, maybe compounding, your efforts”. It’s a question about how you remove both your backlog of toil and resolve the inbound toil that daily changes to software systems represents. It’s a question of “how do you chose to think about your work as you’re doing it?” DevOps is an improvement mindset, not just about code and configuration, but about organizational cohesion, communication, collaboration, and engineering culture.

DevOps Is More than “Developers and Operations”

Unlike those whose staunch opinions limit the scope of DevOps to refer to activities of developers and operations engineers, though managing scope creep is important, the agency and effect of DevOps are clearly intertwined with an organizations capability to also foster a set of core values and principles. For over a decade, we’ve seen some teams going so much faster than others that it doesn’t matter who’s slower, the result out to production is buggy, insecure, and unscaleable systems. This very common and dysfunctional clock speed mismatch also extends to testing, risk and compliance, budgeting (i.e. part of planning), architecture, and proper monitoring/telemetry/measurement practices.

Similarly, an ‘agile’ team find it very hard to be ‘agile’ without the buy-in of leadership, support from finance and legal, alignment with marketing and sales, and deep connection to its customers/stakeholders as well as a whole slew of other examples that underline how agile can’t be just a dev-centric worldview.

I hesitate to invent more terms such as ‘DevBizOps’, ‘DevTestOps’, and ‘DevSecOps’ simply to segment off these augmented versions of DevOps when these areas of concern should be integrated into systems delivery lifecycles, to begin with. I don’t want to conflate concepts, yet so many elements overlap, it’s hard to know how to have one conversation without having three others too. At the end of the day though, it’s as arbitrary to me that people invent new terminology as it is that some others chose to dig their fundamentalist heels in about exclusively keeping the scope of DevOps to two traditionally distinct groups, particularly when we see modern interpretations of DevOps blends skills and responsibilities to the point that we highly value (pay) SREs and “10x developers” because they defy narrowly scoped job opportunities in favor of cross-functional roles.

What are *My* Core Values and Principles of DevOps?

Again, this is a question you should seek to answer in your own way, but after only five years of immersion and purposefully seeking out other perspectives daily, a few key elements seem to hold:

  • Transparency
    • Observability
    • Traceability
    • Open Systems
    • Great Documentation
  • Inclusivity
    • Collaboration
    • Swarming / Blameless Fault Handling
    • Customer/User-focus
    • Prioritization to Value
  • Measurement
    • Building Quality In
    • High-value Feedback Loops
    • Useful Monitoring and Telemetry
    • Clear SLA/SLO/SLIs
  • Improvement
    • Automate AMAP and by default
    • Continuous Learning
    • Coaching and Mentoring
  • Alignment
    • Small Batches, Short-lived Change Cycles
    • Clear Processes (Onboarding, Approvals, Operationalizing, Patching, Disposal)
    • Work Contextualized to Biz Value/Risk
    • All Stakeholders Represented in Decisions

DevOps Is Inclusive by Nature

When dialed in, the values and principles of DevOps foster an environment of high-trust and safety, synthesize perspectives without losing focus, and balance personal and team capacity to compound individual contributions rather than burn out talented professionals. DevOps is not, as Audre Lord (via Kelsey Merkley) puts it, our “master’s tool”, so long as we don’t make it such; rather, if we decide that DevOps must not inherit the meritocracy and exclusivity of corporate management and Agilista methodology, it must be a truly better alternative than that which came before.

This is how DevOps can change things, this is why it must. New thinking and new voices provide a way out of tech monoculture. In order to improve a system, you need more than the one option you’ve already found faulty. This is why I personally support local non-profits like Resilient Coders of Boston. DevOps values and principles offer a far better chance for new voices to be at the table, at least the version I’m advocating.

DevOps Improves Organizations In-Place

Imagine if your teams, your colleagues, your leadership really internalized and implemented the core values and principles defined above. Doing so in finance, human resources, marketing, sales, and even [dare I say] board of trustees groups would cause a natural affinity for supporting systems engineering teams. Conversely too, developers about to change a line of code would be more likely to ask “how does this impact the customer and my organization’s goals?”, “what documentation do customers and sales engineers need to effectively amplify the new feature I’m developing?”, and “how could this process be improved for everyone?”.

There is an old stereotype of programmers being bad at “soft skills”, resulting in miscommunication, disconnect of work from business value, “not my problem” mentality, and throw-it-over-the-fence mindset. My perspective on DevOps is that none of these things would have the room to materialize in organizations that put processes in place to ensure the above values and principals are the norm.

Everyone can do these things, there’s nothing stopping us, unicorn to F100. Importantly, transitioning from what is currently in place to these values takes time. Since time is money, it takes money too. DevOps advocacy isn’t good enough to make the case for these changes and the spend that comes attached. No one developer or even team can change an organization, it takes Gestalt thinking and demonstrated value to get people ‘bought in’ to change.

Facilitating DevOps Buy-In

The first thing to get straight about DevOps is that it takes conversation and active engagement. This is not and never should be something you go to school or get a certificate for; it is a journey more akin to learning a martial art like Uechi-ryu than a course you pay thousands of dollars for at some tech industry Carnivale.

Collect those in your organization is interested in having a conversation about cultural implications of DevOps values and principles, using something lightweight like a lunchtime guild, a book club, or even a Slack channel. Listen to people, what they think and already know, and don’t assume your perspective is more accurate than theirs. Be consistent about when/what you do together to further the dialog, hold a local open spaces meetup (or find someone who does this, like me), and invite people from outside the engineering team scope, such as a VP or member of product management at your organization, and ASK THEM afterwards what they thought.

Once you have people from different levels engaging on similar wavelengths about DevOps, ask them to help each other understand what are some first tangible and tactical steps to improve the current situation based on some of the core values and principles either defined above or further crafted to fit your circumstances. Get a headline name in one or more regional DevOpsDays organizing committees to come visit as an outside perspective to what you’ve got going. And importantly, make time for this improvement. SEV1 incidents aside, there’s always some weekly space on everyone’s calendar that’s an optimal time to get together.

Or you can just ping me [at] paulsbruce [dot] io or on Twitter and we can figure out a good way for you to get started on your own DevOps journey. I’m always happy to help.

On Lack of Transparency in SaaS Providers

As many organizations transition their technical systems to SaaS offerings they don’t own or operate, I find it surprising that when a company acquires a 3rd-party offering deployed on said offerings, they are often told to “just trust us” about security, performance, and scalability. I’m a performance nerd, that and DevOps mindset are my most active areas of work and research, so this perspective is scoped to that topic.

In my experience amongst large organizations and DevOps teams, the “hope is not a strategy” principle seems to be missing in the transition from internal team speak and external service agreement. Inside a 3rd-party vendor, say Salesforce Commerce Cloud, I’m sure they very skilled at what they do (I’m not guessing here, I know folks who work in technical teams in Burlington MA). But even espousing a trust-but-verify culture internally, when your statement to customers who are concerned about performance at scale of your offering is “just trust us”, seems maligned.

TL;DR: SaaS Providers, Improve Your Transparency

If you provide a shared tenancy service that’s based on cloud and I can’t acquire service-level performance, security audits, and error logs that are isolated to my account, it’s a transparent view into how little your internal processes (if they even exist around these concerns) actually improve service for me, your customer.

If you do provide these metrics to internal [product] teams, ask “why do we do that in the first place?” Consider that the same answers you come up with almost always equally apply to those external consumers who pay for your services that are also technologists, have revenue on the line, and care about delivering value successfully with minimal issues across a continuous delivery model.

If you don’t do a good job internally of continuously measuring and synthesizing the importance of performance, security, and error/issue data, please for the love of whatever get on that right now. It helps you, the teams you serve, and ultimately customers to have products and services that are accurate, verifiable, and reliable.

How Do You Move from “Trust Us” to Tangible Outcomes?

Like any good engineer, when a problem is big or ambiguous, start breaking that monolith up. If someone says “trust us”, be specific about what you’re looking to achieve and what you need to do that, which puts the ownness on them to map what they have to your terms. Sometimes this is easy, other times it’s not. Both are useful areas of useful information, what you do know and what you don’t. Then you can double-click into how to unpack unknowns (unknowables) in the new landscape.

For SaaS performance, at a high level we look for:

  • Uptime and availability reports (general) and the frequency of publication
  • Data on latency, the more granular to service or resource the better
  • Throughput (typically in Mbps etc.) for the domains hosted or serviced
  • Error # and/or rate, and if error detail is also provided in the form of logs
  • Queueing or otherwise service ingress congestion
  • Some gauge or measure of usage vs. [account] limits and capacity
  • Failover and balancing events (such as circuit breaks or load balancing changes)

You may be hard-pressed to expect some of these pieces of telemetry provided in real-time from your SaaS provider, but they serve as concrete talking points of what typical performance engineering practices need to verify about systems under load.

Real-world Example: Coaching a National Retailer

A message I sent today to a customer, names omitted:

[Dir of Performance Operations],

As I’m on a call with the IEEE on supplier/acquirer semantics in the context of DevOps, it occurs to me that the key element missing in [Retailer’s] transition from legacy web solution last year to that which is now deployed via Commerce Cloud, the lack of transparency (or simply not asking on our part) over service underpinnings is a significant risk, both in terms of system readiness and unanticipated costs. My work with the standard brought around two ideas in terms of what [Retailer] should expect from Salesforce:

A) what their process is for verifying the readiness of the services and service-level rendered to [Retailer], and

B) demonstrated evidence of what occurs (service levels and failover mechanisms) under significant pressure to their services

In the past, [Retailer’s] performance engineering practice had the agency to both put pressure on your site/services AND importantly how to measure the impact on your infrastructure. The latter is missing in their service offering, which means that if you run tests and the system results don’t meet your satisfaction, the dialog to resolve them with Salesforce lacks minimum-viable technical discussion points on what is specifically going wrong and how to fix it. This will mean sluggish MTTR and potentially synthesizing the expectation of longer feedback cycles into project/test planning.

Because of shared tenancy, you can’t expect them to hand over server logs, service-level measurements, or real-time entry points to their own internal monitoring solutions. Similarly, no engineering-competent service provider can reasonably expect for consumers to “just trust” that an aggregate product-plus-configuration-plus-customizations solution will perform at large scale, particularly when mission-critical verification was in place before fork-lifting your digital front door to Salesforce. We [vendor] see this need for independent verification of COTS all the time across many industries, despite a lack of proof of failure in the past.

My recommendation is that, as a goal of what you started by creating a ticket with them on this topic, we should progressively seek to receive thorough information on points A and B above from a product-level authority (i.e. product team). If that’s via a support or account rep, that’s fine, but it should be adequate for you to be able to ask more informed questions about architectural service limits, balancing, and failover.

//Paul

What Do You Think?

I’m always seeking other perspectives that my own. If you have a story to tell, question, or otherwise augmentation to this post, please do leave a comment. You can also reach out to me on Twitter, LinkedIn, or email [“me” -at– “paulsbruce” –dot- “io”]. My typical SLA for latency is less than 48hrs unless requests are malformed or malicious.

Performance Engineer vs. Tester

A performance engineer’s job is to get things to work really, really well.

Some might say that the difference between being a performance tester and a performance engineer boils down to scope. The scope of a tester is testing, to construct, execute and verify test results. An engineer seeks to understand, validate, and improve the operational context of a system.

Sure, let’s go with that for now, but really the difference is an appetite for curiosity. Some people treat monoliths as something to fear or control. Others explore them, learn how to move beyond them, and how to bring others along in the journey.

Testing Is Just a Necessary Tactic of an Engineer

Imagine being an advisor to a professional musician, their performance engineer. What would that involve? You wouldn’t just administer tests, you would carefully coach, craft instruction, listen and observe, seek counsel from other musicians and advisors, ultimately to provide the best possible path forward to your client. You would need to know their domain, their processes, their talents and weaknesses, their struggle.

With software teams and complex distributed systems, a lot can go wrong very quickly. Everyone tends to assume their best intentions manifest into their code, that what they build is today’s best. Then time goes by and everything more than 6 months old is already brownfield. What if the design of a thing is already so riddled with false assumptions and unknowns that everything is brownfield before it even begins.

Pretend with me for a moment, that if you were to embody the software you write, become your code, and look at your operational lifecycle as if it was your binary career, your future would be a bleak landscape of retirement options. Your code has a half-life.

Everything Is Flawed from the Moment of Inception

Most software is like this…not complete shit but more like well-intentioned gift baskets full of fruits, candies, pretty things, easter eggs, and bunny droppings. Spoils the whole fucking lot when you find them in there. A session management microservice that only starts to lose sessions once a few hundred people are active. An obese 3mb CSS file accidentally included in the final deployment. A reindexing process that tanks your order fulfillment process to 45 seconds, giving customers just enough time to rethink.

Performance engineer doesn’t simply polish turds. We help people not to build broken systems to begin with. In planning meetings, we coach people to ask critical performance questions by asking those questions in a way that appeals to their ego and curiosity at a time that’s cost effective to do so. We write in BIG BOLD RED SHARPIE in a corner of the sprint board what the percentage slow-down to the login process the nightly build as now caused. We develop an easy way to assess the performance of changes and new code, so that task templates in JIRA can include the “performance checkbox” in a meaningful way with simple steps on a wiki page.

Engineers Ask Questions Because Curiosity Is Their Skill

We ask how a young SRE’s good intentions of wrapping u statistical R models from a data sciences product team in Docker containers to speed deployment to production will affect resources, how they intend on measuring the change impact so that the CFO isn’t going to be knocking down their door the next day.

We ask why the architects didn’t impose requirements on their GraphQL queries to deliver only the fields necessary within JSON responses to mobile app clients, so that developers aren’t even allowed to reinvent the ‘SELECT * FROM’ mistake so rampant in legacy relational and OLAP systems.

We ask what the appropriate limits should be to auto-scaling and load balancing strategies and when we’d like to be alerted that our instance limits and contractual bandwidth limits are approaching cutoff levels. We provide cross-domain expertise from Ops, Dev, and Test to continuously integrate the evidence of false assumptions back into the earliest cycle possible. There should be processes in place to expose and capture things which can’t always be known at the time of planning.

Testers ask questions (or should) before they start testing. Entry/exit criteria, requirements gathering, test data, branch coverage expectations, results format, sure. Testing is important but is only a tactic.

Engineers Improve Process, Systems, and Teams

In contrast, engineering has the curiosity and the expertise to get ahead of testing so that when it comes time, the only surprises are the ones that are actually surprising, those problems that no one could have anticipated, and to advise on how to solve them based on evidence and team feedbacks collected throughout planning, implementation, and operation cycles.

An engineer’s greatest hope is to make things work really, really well. That hope extends beyond the software, the hardware, and the environment. It includes the teams, the processes, the business risks, and the end-user expectations.

Three, Sixty, Five

In less than three days, I recently passed through five airports using six planes to meet with three very important teams. Not a single Uber (because fuck Uber), 14 meetings/calls later, and a collective 9 hours of sleep. Before I let the normal hurricane of work-life sweep these events away, I want to distill learnings.

First…Context:

In 2017 after an exhaustive job search, I found myself talking with an SMB CEO about what my goal was. I said then, as I still say now, that one of my primary goals is to “be in the room”. Rooms where decisions are made; not only internal decisions, external ones too. You have to prove your value at the table in these rooms, and this is what I do.

Three

During these past three days, three very meaningful things happened.

One: an F100 customer I’ve invested a ton of effort in for the past 6 months demonstrated how impactful that work has been to their strategic quarterly planning in front of two very important people in the full-time company I work with. This took me by surprise, but is exactly what happens when you’re doing things right (I think). People benefit so much they want to share it with others. I see this as a positive characteristic of humanity.

Two: another F100 prospect I’ve invested a ton of time with confidentially shared how the internal dynamics of their big-picture financial strategy over technology funding underscores the importance of having a flexible approach to the acquisition process for software vendors and GSI partnerships. I would never have had the opportunity to learn this in school, at prior software jobs, or without the support of an amazing business team (incl. key individuals in leadership, partner channel, sales, and product management).

Three: I got another very private F100 customer to discuss patterns, practices, and industry trends with a senior researcher at a major analyst firm. Early in my career and for many people in what I refer to as ‘shallow engagement work patterns’, this would be impossible; but I believe in value-first relationships, referring to how Tim O’Reilly puts it: “create more value than you consume”. Really doing this with your customers takes more than an ask, it takes gives which make asks pale in comparison.

Sixty

These and other things happened the past almost-three days, but took sustained efforts. The same statement applies every day inside my day job. Some things happen overnight. Some things aren’t under your control. That leaves the universe of possibility to things that happen over time and are totally under your control. This is very good news for us “busy brains”.

Self-control is a characteristic people pick up on subconsciously, but lack thereof most certainly triggers conscious flags. I have a few habits…nail-bitten fingertips, post-its written everywhere, late-night emails…that I think qualify for the latter. In contrast to the last sixty hours, I’m going to scale back these and other habits over the next sixty days. This will take effort, sustained and supported with the help of others.

A colleague said to me last month: “don’t wait for your doctor to tell you”. Habits are something that can be changed. Trust takes time to build and can be lost overnight. Observing people’s habits can accelerate identifying trustworthiness. Being someone I know is trustworthy, I want to seek ways to accelerate others’ trust in me. Controlling habits is my immediate actionable.

Five

If you’re ambitious, curious, and capable, tech is a fantastic sector to be in. Things move very fast, money too. Five years ago, I very deliberately pivoted my career from writing code all day to observing and directing the code that people operate by. In retro, I could have done this in my early twenties, but at least it’s happening now.

Since 2018, the rooms I’ve gotten really good at working in are typically at an F100. Under the surface, things in F100s are so complicated and fucking messy…I love it when things are messy…but on the shiny topside in rooms with egos and initiatives and collateral on the line, dialogs have to be squeaky clean. The trick is how to bring the room together with anything but the mundane, with active listening, informed point of view, inspiring next-challenges, and most importantly, sincerity. Effective communication and collaboration takes experience, and experience takes doing. This is what I’m doing all the time now, honing and refining.

Holiday IoT and the Performance Imperative

A few words to manufacturers and vendors of tech toys: to really be ready for the holiday, if your product requires software updates in order to work or is in any way internet connected, make sure your site stays up. Otherwise, you just shipped coal.

  • Provide more than one distribution point for downloadable updates/binaries
  • Rely on CDNs for static assets (like installers and documentation)
  • If your update process must rely on live services, make sure they’re scalable
    • Load test subcomponents/microservices AND the end-to-end process
  • Be prepared for damage control by:
    • Monitoring site uptime and availability to know when things are broken
    • Proactively establish a communication channel with customers during issues
    • Properly staff IT and support for during AND post-season issues
  • Make sure the cost of downtime is factored into your next sales cycle

Santa Brought Us a Brick?

Let me start by admitting how 1st-world this example is. Robots as play-things are still not exactly ‘so easy, a child could do it’, and Roomba’s have been around for almost two decades, but we still have yet to see a really down-to-earth home robotics project that really works for children under 10. I don’t just mean toys that are not pre-assembled, even the right-out-of-the-box kind often require firmware updates or online services to really work as expected.

Case in point, the Meccano MAX. Though it only took about 3hrs total to put together, this morning when we finally turned it on and went to connect for a firmware update, the vendor’s website was down…hard. The instructions said, before anything else, update the ‘MeccaMind’ and voice commands weren’t working without, so, blocker.

As an ops nerd, I slapped an uptime monitor on it to know when (if ever) it was back up:

That didn’t stop the whole multi-day experience from deflating to a dud. We all worked on this thing together and then before it can do anything, we are stuck guessing about when we can actually enjoy it. Don’t blame Santa, blame the geniuses at Meccano.

Performance, of your product, of your service, of your site, is imperative to delivering what you sold people. Availability, uptime, scalability, and reliability matter by default now. Everyone has downtime, but 4hr recovery time on your corporate domain isn’t just irresponsible and costly, it’s plain embarrassing…and transparent.

Why Is This Even Important?

My 7yr old is crazy into coding right now. Granted, we use a visual Code Block Editor mostly for lights and tones, but it’s a great way to introduce concepts like flow control and formal logic. As soon as she saw an example I built that used function blocks to encapsulate and reuse logic, she instantly understood and started refactoring her programs.

But finding the right project for varying stages of aptitude, appetite, and enjoyment is a real challenge. No thanks to marketing, but also unanticipated road-blocks like service and subscription dependencies are hard for consumers to factor in when purchasing. Even when you do find a right-fit project, if bone-headed problems like website downtime occur, it can become a negative experience for the child (or student).

It’s important for STEM product manufacturers and software vendors to really think about the impact of what they’re selling, how they’re delivering it, and how to support people who paid them money for something to accomplish a goal. If you don’t have the optimal consumer’s experience in mind, it will eventually cost you.

Need the Robot Software Updater?

I can’t archive everything on their site, and it’s their job to provide reliable content distribution, but in case you find yourself stuck like I was, here are links to at least the firmware updater tool:

Also, never ask a consumer if they want to choose (null).

And if the updater gives you flashbacks to DirectX drivers from 1997, don’t worry. It only looked like it bricked my robot for about 4mins before providing UI feedback:


Once you do get back to the modern era, a 98.6MB mobile app to control it shouldn’t be too hard on your data plan. They also need to know your GPS location, phone contacts, and file storage for some reason.

Value Chain in DevOps

Foreward: Since I highly doubt the following concepts will see the light of day in the final draft of IEEE 2675, I wanted to document that in fact I pushed this to the group June 15th 2017. During a subsequent review, it got huge push-back from our Curator in Chief, the early first of a future string of events that lead me to publish this primary work on my personal blog.

What is a ‘value chain’?

value chain is a set of activities that a firm operating in a specific industry performs in order to deliver a valuable product or service for the market. The concept comes through business management and was first described by Michael Porter in his 1985 publication, Competitive Advantage: Creating and Sustaining Superior Performance.[1]

The idea of the value chain is based on the process view of organizations, the idea of seeing a manufacturing (or service) organization as a system, made up of subsystems each with inputs, transformation processes and outputs. Inputs, transformation processes, and outputs involve the acquisition and consumption of resources – money, labor, materials, equipment, buildings, land, administration and management. How value chain activities are carried out determines costs and affects profits.— IfM, Cambridge[2]

[Wikipedia: Value Chain (Porter)]

How does it related to IEEE 2675?

As related to DevOps, the Value Chain for Software Delivery is an application of a lifecycle perspective that scopes standards adherence to only certain individuals based on their participation in primary or supporting activities related to a particular software product.

DevOps is about continuously delivering value to users/customers/consumers. A value chain perspective disaggregates activities from an organizational funnel, making it easier for teams and consumers of a particular product or service to ask “does this thing meet the standard” without unintentionally including other unrelated teams or products, were the question be phrased as “does this organization meet the standard”.

What problem are we solving by using ‘value chain’?

Adoption. In large organizations with many independent groups or product teams, DevOps principals may apply across the value chain of one product, but not another, provided these products are completely independent from one another. For an organization or team to claim that a particular software product adheres to this standard, all aspects of that product’s value chain must implement the principals and practices set forth in this standard.

Examples of where the broadness of using “organization” presents a challenge to adoption:

  • Consulting agencies with many independent project teams on working on separate contracts/products
    • If only one of those contacts require adherence to 2675, does this require every team/contract (both in the future and retroactively) to do the same?
    • Using “value chain” would scope 2675 adherence to any and all parties performing activities germane to delivering that specific contract/product
    • We know that if even one team implements DevOps per 2675 and sees success, organizations are likely to grow that out to other teams over time; “value chain” helps adoption of the standard. 

  • Enterprises in the midst of transformation to DevOps
    • Can they claim adherence on a specific product if the “whole organization” can’t yet?
    • Much like the above agencies argument, when the scope of adherence is based on activities relating to delivery of a project, enterprises are far more capable of becoming “DevOps ready” because they can grow the practice out *over time*
    • In DevOps, key success factors are determined – driven – by customers, the end user.

  • Organization’s requirements from non-technical internal agencies
    • Can we expect that the legal or HR departments are also “DevOps”? 
      This would of course need to be defined by activities that support the delivery side of the business (i.e. billing, purchasing, etc.), begetting an activities-based perspective on implementation over organizational labeling

Why is this of importance to IEEE 2675?

In layman’s terms, adoption of IEEE 2675 at an organizational level can’t happen overnight, especially in large enterprises with many teams. Fortunately, it doesn’t have to, provided we adequately scope ‘shall’ statements with a perspective that A) is reasonable in scope and impact on the org, and B) enables parties to agree on what it means for a software product to have been developed and delivered using this standard.

How and where would we use ‘value chain’?

Places in the text where use of the word ‘organization’ could infer that IEEE 2675 must be implemented across the whole organization before any single team or product could claim adherence to the standard. For instance:

  • Too broad:

    “Organizations shall implement effective continuous delivery procedures aligned with the architecture definition in a manner that meets the business needs of the system.” … “DevOps itself requires effective architecture across the organization to ensure that application build, package and deployment procedures are implemented in a robust and consistent manner.” (6.4.4)

  • Scoped: 

    “Organizations shall implement effective continuous delivery procedures within a particular value chain that are aligned with the architecture definition and in a manner that meets the business needs of the system.” … “DevOps itself requires effective architecture across the whole value chain to ensure that application build, package and deployment procedures are implemented in a robust and consistent manner.”

  • Too broad: 

    “Organizations shall maintain an accurate record of both code deployed and fully automated mechanisms in order to remove obsolete code.”

  • Scoped:

    “Organizations shall maintain an accurate record of both code deployed and fully automated mechanisms in a value chain in order to remove obsolete code.”

Additional references:

Wanting vs. Having for My 7yr Old

I make a lot of mistakes. I try not to do it on Github where commits are a permanent record of your competencies. So are children, only the impact of a word said harshly or missed opportunity for positive reinforcement take immediate effect and often have lasting effects on their lives.

Taking my son and daughter out one-on-one to get gifts for the people in their lives is important to me, and despite sometimes feeling like it in the moment, is never a mistake in the ideal sense. Having a positive experience doing so is also part of a successful outing, because if it’s a negative experience, the whole thing is useless.

A few weekends ago, I went out with my daughter. When we finally ended up at the toy store to get something for her brother, naturally she found something for herself. Because she’s seven, and no matter how zoned in of a quick pow-pow we had outside before entering about who she’s here to shop for, she still looks at the world with her-eyes. Arguably we all do this, regardless of our age, defaulting to looking out for number one in all manner of situations. That day, it was a cheap, pink, plastic set of glam rings.

I said no, but I’ll think about it, and asked her to put them back and focus on her brother’s gift. What a mistake. She didn’t throw a fit or get weepy or shut down. I took a picture (for when I would come back in search of stocking stuffers) and moved on. The Big Day is a week away, and I realize there was a missed opportunity there and I have to find a moment with her when she gets them in her stocking to make this memory-point:

In life, the moment you want a thing and the moment you get it are often different.

If I had bought her that thing, what next? Should she have it right then, as inevitably she would. This is just kicking the ‘delayed gratification’ can down the road. And in juxtaposition with the notion of immediate positive reinforcements, her want wasn’t connected to a prior goal or reward plan. So no, I don’t just buy things because my kids want them, even when I can. It’s not just the principle that matters, my choices (a.k.a. commits to their codebase) have an immediate impact. The manner in which I roll those decisions out matters as much as the very moment of desire in their hearts.

So what will I do with this new hypothesis? As with all hypotheses, I will test it. On the morning of the Big Day, after all the things are unwrapped and people chill the fark back down and people have 2nd coffees and there’s bandwidth for sharing that learning, I will say to her:

“Love, I too seek the desires of your heart and will always support you in them. In life, the moment you want a thing and the moment you get it are often different times. Remember this thing, what you wanted weeks ago? Well I remembered and it’s been waiting for you the whole time. But there’s a time and a place, which is today! Trust in me, one who has gone before, that I can help you learn to deal with the waiting time between when you desire something and when you achieve it. Remember that my love for you is something you don’t always know how to see, but never doubt that it will always be there.”

AllDayDevOps 2018: Progressive Testing to Meet the Performance Imperative

Mostly as an appendix for references and readings, but whenever I can I like to have a self-hosted post to link everything back to about a particular presentation.

My slides for the presy: https://docs.google.com/presentation/d/1OpniWRDgdbXSTqSs8g4ofXwRpE78RPjLmTkJX03o0Gg/edit?usp=sharing

Video stream: https://www.youtube.com/watch?v=DexfpnbBFn8

A few thoughts from my journal today (spelling and grammar checks off):

Will update as the conversation unfolds.