September 13, 2019

A developer goes to a DevOps conference

Posted in Software at 18:54 by graham

As a professional software engineer and enthusiastic pro-am systems administrator, I have long been curious about DevOps.

I spent the last two days at PDX DevOpsDays. Across organised talks, Ignite’s, Open Spaces, and as many hallway conversations as I could fit in, here is what I learned about what “DevOps” is and does.

DevOps is Ops with new tools. I went to DevOpsDays thinking DevOps means developers and operations merged into one team. I was wrong.

I did not meet or hear from any developers. It was a gathering of system administrators who use version control, and write a lot of YAML. Programming languages were not a topic; I heard Python mentioned once in passing, that’s it.

DevOps means the veteran admins had to check in their personal scripts, and everyone is expected to automate more things. They relate to the software their business runs (“app”, “asset”) the way a merchant-navy captain relates to what’s in the containers on his ship.

AWS is king. Whenever a hosting provider was mentioned, it was AWS. It was usually mentioned as if it was a basic understanding that we’re both on AWS. Google Cloud was a sponsor but didn’t otherwise get mentioned, Digital Ocean was mentioned once, and one of my lunch tables had someone who uses Azure. Otherwise it was all AWS.

Slack is king. “Chatops” is big. Being a gathering of system administrators I’d expected more IRC, there was no mention of it.

Everyone hates YAML. Everyone writes a lot of YAML.

On-call / incident-response is a big part of the DevOps job, and several of the sessions focused solely on that. As a result they also talk about mental health more than programmers do.

Cloud monitoring is a saturated market. Maybe half the vendors/sponsors were cloud monitoring (i.e. Prometheus competitors), although most of them called it something else.

Kubernetes, Terraform, Ansible and Packer were the tools I heard mentioned the most, in that order. Prometheus was popular for monitoring, but many (most?) used a vendor, such as Datadog.

The most common job title seemed to be SRE (Site Reliability Engineer), although there was a long tail, and they don’t care much about job titles.

Many were curious about service-meshes (basically a smart network proxy?) such as Istio or Linkerd, but almost none were using one.

The consensus on how software releases happen is that your CI/CD pipeline builds and tests the “asset” resulting in a Docker container, which is they deployed in a blue-green fashion using either Kubernetes or Terraform. The existence of a CI/CD pipeline that produces a Docker container was assumed in most conversations I had. Everyone loves containers.

PDX DevOpsDays was one of the best run conferences I’ve been to (MC Alice Goldfuss was phenomenal), and probably the one I learnt the most at. A++++ would conference again.

13 Comments »

  1. Juhani said,

    September 29, 2019 at 14:58

    Devops feels similar to design for manufacturing, for automated factories.

    For dev, it seems to contain the other half of product lifecycle, when theory meets real life messiness, live systems, 24×7, metrics needed for functionality, failures, long term sustainability, repairability and archival, not just functionality. For ops, devops seems to contain more abstraction, reproducible results, enforced code review, though not yet that much of automated testing; no more sloppy bash cron jobs on some sysadmin-s personal home directory and under his credentials. Still ops does not care that much about business functionality, still no solutions with timed self-destruct (ops or dev could create some reports that are needed only once/twice), still very far from business people.

    In manufacturing the cost saving method with the largest impact is design for manufacturing. Design for manufacturing in electronics means reduced amount of different components, testing options, no flipping of the board and similar.

    Reminds me of a story about E.Musk. He was working on an industrial robot, asked a nearby man: this does not work, are you responsible for this robot? The man answered: responsible for configuring the robot or designing the detail? The man was fired the next day. Although the man seemed to have a valid question – it’s overly complex to automate things that are not designed to be built using automation. Curse of reductionism.

  2. Vaidik Kapoor said,

    September 29, 2019 at 13:57

    I agree with Amor Shapira about the sense you got about DevOps. DevOps is about collaboration between devs and ops towards building software and operating it efficiently to meet business needs. Traditionally building and operating were looked very differently but experience has taught us that features and stability cannot be achieved without understanding of how features were built and what needs to be done to run those features in production efficiently. In an ideal world, the same person should be able to build and operate systems in production however we don’t live in an ideal world – traditionally there was a big divide between devs and ops, and in complex environments you can’t learn to do everything yourself. Hence… DevOps.

    There are multiple ways this can be achieved. This article is a good read to understand different models of implementation of DevOps culture – https://web.devopstopologies.com/

    “DevOps is Ops with new tools. I went to DevOpsDays thinking DevOps means developers and operations merged into one team. I was wrong.” – This is kind of right but not completely. DevOps is about bringing ops closer to dev teams by either merging ops or retraining devs to be able to be better at ops (there are more ways as indicated in the above article about DevOps topologies). The idea of doing ops with new tools is that these new tools help us with doing ops like we are building software. Many software development best practices, abstractions, architectures can be now applied to ops as well. This shift in doing ops with these tools which are a lot like other software development tools enable ops to come closer to devs and vice-verse. Here is a great talk by Mitchell Hashimoto (the creator of Terraform, Packer, Vagrant, Vault, etc.) that got me into DevOps – https://www.youtube.com/watch?v=UTQQggVx4sI. This talk explains the use of tools with the backdrop of DevOps.

    Other than this, your observation was spot on. I was not at the event but that’s the story at most DevOps events.

    I think software architecture will soon become an area of focus in DevOps circles as the right architecture also essential for achieving CI/CD, agility and DevOps.

    Never the less, welcome to the world of DevOps. :)

  3. Scott Kinney said,

    September 29, 2019 at 13:48

    Haha, yes. I believe this. My title is “DevOps Engineer” but I think “Cloud Ops” is more appropriate.

  4. Sumit Khanna said,

    September 29, 2019 at 04:19

    A developer must keep these bare minimal devops know-hows today. Very superficial peek into life as devops, but I had been expecting/recommend the ELK stack, with APM/Elastalert too. Also, YAMLs are lovely and I wonder why a good devops ‘d hate it.

  5. owen said,

    September 29, 2019 at 02:50

    well in either case at least we will get some awesome new systems and solutions out of it. Like rocketships and flying cars and stuff rather than just changing stuff for changing stuff sake. Like if I deploy a database on docker use it for 15 years straight what do I do when something goes wrong? re-deploy it fresh after 15 years?

  6. Stephen said,

    September 29, 2019 at 01:06

    Hrm. I suppose that’s why my interviews for devops hasn’t been working too well.

  7. Ahmet San said,

    September 28, 2019 at 19:49

    Until developers assume ops responsibilities in their code, devops will not realize it’s promised potential – that automation and scripting must be a core part of the code that runs in prod… My first question to developers: do you get paged? If not, you still need ops people regardless of how your operation defines nomenclature

  8. dingdong said,

    September 28, 2019 at 18:41

    feels like a lot of these points derive from being an in-person convention held in north america, as opposed to more distributed conventions or those held outside of north america. especially wrt aws, slack, specific tools, lack of programming language involvement, lack of dev integration

  9. Mike said,

    September 23, 2019 at 04:37

    Our ops team, or more specifically their director talk a lot about DevOps. He puts a lot of faith into it, silver bullet style. Our actual dev team never talk about it, probably don’t even know what it means. Ops team is too busy manually installing artifacts/ assemblies from the dev team onto servers.

  10. Liz Bu said,

    September 20, 2019 at 00:25

    Have you seen new tools like Pulumi, that let you use real programming languages (like Python) and eliminate the YAML? I’ve tried Pulumi for the last few months and it’s been great!

  11. Amos Shapira said,

    September 17, 2019 at 11:28

    Great summary.

    Not to discard your experience but the following:

    “DevOps means the veteran admins had to check in their personal scripts, and everyone is expected to automate more things. They relate to the software their business runs (“app”, “asset”) the way a merchant-navy captain relates to what’s in the containers on his ship.”

    Makes me feel like something is missing in your impression. To me, “DevOps” really does mean a mesh of developers and operations roles into one team where developers and “ops” guys work together on making the application code take into account the operational requirements (e.g. it reads its configuration in a way that gels well with the way the configuration is managed, it understands and cooperates with redundancy and high availability requirements, etc). The “ops” people in the organization provide the platforms and guidelines/best practices on top of which the developers supply the application code.

  12. David Evans said,

    September 16, 2019 at 22:00

    As a fellow Dev I’ve also been really curious about DevOps, and it feels like the term “Agile” has been used on the Dev side much more to talk about related topics of resilience and fast feedback. When I mention DevOps in my circles it also only seems to be about the operations side.

  13. @realdancollins said,

    September 13, 2019 at 22:31

    Welcome to DevOps, Graham! Great blog post! And good for you for jumping in and trying to add value. One of the most important pieces of DevOps is feedback and these kinds of posts are super helpful to anyone interested in leading the community.

Leave a Comment

Note: Your comment will only appear on the site once I approve it manually. This can take a day or two. Thanks for taking the time to comment.