270
Reddit DevOps. #devops Thanks @reddit2telegram and @r_channels
How are teams keeping docs current when you ship several times a week?
We ship several times a week and keeping the docs in sync is getting harder, code changes have a clear review process, docs don’t. Someone changes a parameter or response field, the release goes out, then a few weeks later we find examples still using the old version. We tried adding docs to the PR checklist, but engineers don’t always know which pages a change affects, what’s worked for keeping docs current? Would love any suggestions
https://redd.it/1wk6vpl
@r_devops
Github struggling under high PR throughput?
I'm on a 3 person team where we are each shipping 40-50 PRs a day (super early stage company so we're optimizing for getting a functional product out quickly).
As we've ramped up the velocity things downstream have begun to show cracks. For example, we were requiring PRs to be up to date with main before merge, but at our scale the PRs were constantly getting stale which was leaving us blocked for long stretches.
We've ended up cutting pre-merge CI way down and batching more testing after merge, which has made things "better" but feels backwards.
Curious if anyone else run into similar issues? Did you end up finding a better solution?
https://redd.it/1wjtf3u
@r_devops
RANT: Has anyone experienced vibe planning?
This is a rant, my emotions are still fresh - so I'm coming at this one sided. We just wrapped up PI planning our current planning session is mess with a lower confidence vote. Story points are all over the place with no rhyme or reason.
Our PO and Directors vibe-planned our features and stories and then dropped them on us to refine. Looking through the stories some of this stuff just doesn't make sense and doesn't exactly apply in our situation. There are so many assumptions being made. But boy does it look pretty because of the hallucinated architecture diagrams. Features were proposed and accepted by the VPs and praised at how well thought out the feature description and documentation was.
I made my first pass on a feature and then poof the stories are gone. The PO had Copilot go through with updated information and it decided to make a new feature and move all the stories over halfway through our PI planning meeting. "It's fine, just have your copilot refine it with your details with additional context, it shouldn't take long". Our scrum master is getting after us because our stories weren't refined by the end of the meeting. I just want my stores to make sense and executable. Just frustrating.
I use Copilot like crazy for coding, but for planning features and stories? You ask a question to the PO and they can't answer on the top of their head. They have to read the feature because they didn't write it.
https://redd.it/1wjqrl3
@r_devops
Startup idea validation
Hey everyone
We are a team of engineers looking to start a startup around providing infra services so as we are at ideation phase i want to know .
What is the pain point u all face as a devops engineer ?
What do u think u or your organisation will pay for ?
(any specific kind of tool or service)
We are also open for ai integration if that provides value to our product .
This will be extremely helpful to us
https://redd.it/1wjgoo5
@r_devops
What’s the worst in-house DevOps tool you built that you wish a SaaS company would replace?
Every engineering team has that one dark secret:
Three years ago, someone said, "Why pay $300/month for a tool when I can just whip up a bash script, a Dockerfile, and a quick GitHub Action over the weekend?"
Fast-forward to today:
That engineer left the company two years ago.
The script has grown into a 4,000-line shell/Python monstrosity held together by duct tape, raw cron jobs, and unversioned IAM permissions.
You are now stuck acting as an unpaid, 24/7 helpdesk because a product dev’s build failed and they refuse to read the CloudWatch logs.
Upstream AWS deprecated an API, Terraform has drifted into another dimension, and you’re burning half your sprint debugging an in-house tool that generates $0 in business value.
So, let's air some dirty laundry:
1. What is the single most annoying piece of in-house "DevOps glue" your team built that you deeply regret maintaining? (Ephemeral preview environments? Database migration pipelines? Secret syncing? Custom ECS/EKS rollout wrappers?)
2. Why didn't an existing off-the-shelf vendor solve it for you? (Too expensive? Demanded root IAM keys? Over-engineered Kubernetes garbage? Or just broke on edge cases?)
3. If a lean tool solved this exact headache inside your own cloud account without charging your firstborn child, what would it actually take for you to throw a corporate credit card at it?
Tell me your worst build-vs-buy war stories.
https://redd.it/1wje1wz
@r_devops
The best path for Devops engineer
Hi ,I am 4th year engineering student and I am interested in devops engineering and I want the best path for Devops and what's the best ways to learn it and what the best YouTubers to learn from them
https://redd.it/1wiwqmb
@r_devops
how do you benchmarks AI Eval and tracing tools?
i need inputs on this, my company recently had demos from multiple vendors for AI eval and tracing demo, and i realised a lot of them are sharing similar features and capabilities.
but my mind couldn't get out of the fact that why don't have benchmarks for these, if some of you might remember we would have a lot of folks and vendors earlier publishing benchmarks about their tool handling 1 -10 B of logs and traces better than others and querying speed being faster x times, i don't see any of these materials for this category of tools, can any one help me decipher that or is it just that we haven't reached that maturity in the space to do so.
most of the content is just vendors putting things around with feature comparisons nothing else.
https://redd.it/1wj2ylx
@r_devops
Best Devops Certification for Beginners
Hi All,
While I do not have a technical background, my new role as a Project Manager in a DevSecOps organization necessitates that I acquire certifications in both DevOps and cloud computing.
Could you please recommend suitable certification programs for me to pursue?
https://redd.it/1wiwa14
@r_devops
Migrating existing CloudFormation resources to OpenTofu -recommended approach?
Hi everyone,
We’re planning to migrate a large number of existing AWS resources currently managed through CloudFormation to OpenTofu.
Our environments are fairly siloed per customer, and each environment can include resources such as EC2 instances, load balancers, Auto Scaling Groups, VPCs, security groups, and other AWS services.
I’m looking for recommendations from anyone who has gone through a similar migration.
Specifically:
\- Are there any tools that can help convert or migrate existing CloudFormation stacks to OpenTofu?
\- What is the best approach for importing existing resources into OpenTofu state?
\- How did you handle dependencies between resources during the migration?
\- Would you recommend migrating one CloudFormation stack/customer environment at a time?
\- Are there any common pitfalls or state-management issues we should watch out for?
\- What approach would minimize the risk of OpenTofu trying to modify or recreate existing production resources after import?
Our main priority is to make the migration as safe and predictable as possible, with minimal disruption to existing customer environments.
Would appreciate hearing about any tools, workflows, or lessons learned from real-world migrations.
https://redd.it/1witrgz
@r_devops
Is this a solid monorepo or just folders in one repo?
https://redd.it/1wijst2
@r_devops
Vendor management at small companies, is it just "someone senior remembers"?
Specifically, I am talking about small companies where, generally, you won't have a procurement team, and individual teams own vendor management. How do you guys manage vendors?
In my previous stint, I was leading a small DevOps team at a smallish SaaS company, and over a period of a couple of months, we saw multiple issues stemming from software vendors. A blocked card caused payment failures across multiple vendors. A vendor dropped an email setting a deprecation timeline for an API, and we found out by chance at the last moment, causing a last minute scramble to move off the said API on our end. SaaS quota exhaustion brought down prod workflows multiple times.
None of these were vendor outages as such. These were all things we had been told about, or could have been, and we just did not catch them in time.
I am going down a bit of a rabbit hole trying to find the "best practices" around vendor management, if we can call it that?
How do you even handle this? Is there an actual practice here, or is it mostly "someone senior remembers," or do we detect it after it fails?
https://redd.it/1wi18nk
@r_devops
Preserving YAML comments and Git Blame during bulk Kubernetes API migrations (1.16 → 1.32+)
*Affiliation disclosure: I am an open-source maintainer working on Kubernetes migration tooling. This post is a technical breakdown of how we approached AST manipulation, Helm template preservation, and Git blame history.*
Whenever a Kubernetes version deprecates or removes APIs (`extensions/v1beta1` Ingresses in 1.22, `batch/v1beta1` CronJobs in 1.25, `autoscaling/v2beta2` in 1.26), DevOps teams face a recurring dilemma:
1. **Regex / string replacement:** Works for trivial `apiVersion` lines, but fails when schemas change structurally (e.g., adding `spec.selector` in Deployments, or restructuring Ingress paths/services).
2. **`kubectl-convert`:** Parses YAML into Go structs and re-encodes, which **strips 100% of human comments** and mangles custom indentation.
3. **Pluto:** Excellent audit tool, but purely read-only—it won't fix anything in your GitOps repos.
4. **Git Blame Pollution:** When an automated script touches 300 manifests, `git blame` attributes every line to the bot or engineer running the migration, destroying institutional context.
\### How to Address These in an AST Engine
\#### 1. Preserving Comments with CST/AST Parsing
Instead of deserializing into plain dictionaries or JSON, parsing manifests into a Concrete Syntax Tree (CST) using YAML AST nodes allows targeting specific scalar tokens (like `apiVersion: batch/v1beta1` → `batch/v1`) while leaving surrounding comment tokens, blank lines, and indentation untouched.
\#### 2. Unrendered Helm Templates
A common issue in GitOps repositories is storing Helm templates in source control. If you run `helm template` and fix the rendered output, you lose template logic. Building a lexer stage that temporarily masks Go template tags (`{{ ... }}`) as inert string placeholders during AST traversal allows performing the API migration and re-injecting the exact Go expressions byte-for-byte.
\#### 3. Protecting Git History with `.git-blame-ignore-revs`
Git 2.23+ and GitHub natively support ignoring bulk reformatting/migration commits:
```sh
\# Storing the migration commit hash
git log -1 --format="%H # automated k8s migration" >> .git-blame-ignore-revs
git config blame.ignoreRevsFile .git-blame-ignore-revs
https://redd.it/1whx3fm
@r_devops
Devops Job in 2026
Just be frank. Whats is the ideal way to get a Devops Job in 2026. Applying through linkedin naukri indeed doesn’t seem successful.
https://redd.it/1whptst
@r_devops
Stupid question…but what is our job?
Just a bit more context behind the questions what actually is our job/what is honestly expected from us?
Here is a quick back story no actual relative experience, went from graduating from college in a social studies/econ degree, to doing an online masters to geek squad, to help desk to DevOps. been in this field for roughly 6 years now, and most of my learning has been from lurking on DevOps sub/discord, ai, and trial by fire tbh.
My current job seems to have us using so many different tools and expectations such as tracking certs(even app certs), connection issues, resolving security issues, and I mean the list goes on. I thought the primary job for the role was assisting and making code deployment efficient and robust as possible (Ik there is a number of ways to do this). But honestly I was told when I first started the job it was a catch 22 and and I’ll be like a sys admin. But after years of trial by fire I still feel like I’m learning which I’m okay with, but the requirements and expectations from the role seems to be expanding. I here things like platform engineer and now ai engeer (a role designed to guide ai to assist with infra deployments) where is this role going and what is our actual job or are we auppise to branch off at this point?
https://redd.it/1whiujy
@r_devops
Almost giving up on Cloud/DevOps
I have been trying to land my first Cloud/DevOps role for 3yrs to no avail in Canada. I have 4AWS certs , RHCSA and plenty projects. Built core skills in technical support, AWS, Linux, Terraform, Observability. Highly optimized LinkedIn, posting regularly, with experience in sales, level 1 and enterprise SaaS technical support, and I have had like just 4-5 interviews in this field in 3years.
I will be attending AWS re:Invent at the end of the year but I am literally already giving up on the dream.
The worst is that after learning skills and carrying out projects without landing a role, I tend to forget them. And not only that but there's always a tool you feel you ought to learn before you can land a role. This field is extremely competitive and I think the best chance is some promotion within a company that is willing to, otherwise the chances are infinitely slim.
Tbh I'm tired of this rat race and have been unemployed for the whole year. With the number of experienced people being laid off I think it's best for me to change direction and do something else. I'm trying to find another support role and specialize in something else. Maybe take a Masters Degree 6months-1year into my next support role in something with better upside.
Sorry for the rant but this is my story, and I have seen countless other serious people give up after trying for long also.
https://redd.it/1whf5lh
@r_devops
Suggestions / Support Needed
Hi Champions, I’m having 5yoe as a DevOps engineer and in my early career I was working mostly with service based companies. It gave me pretty much experience and foundational understanding of each stack deployments, pipelines, clouds and more. From last 2 years I am working with on prem enterprise products and worked on unmanaged k8 along with managed ones like EKS, GKE and OKE.
Now the problem is:
I recently joined a product base company and their architecture is based on aws managed by TF. The real challenge for me is everything is being managed by HELM and I don’t have much experience with helm charts and sub-charts and I have no idea where to start….
Need suggestions from where I can start leaning this, which can give me the understanding.
https://redd.it/1wjxduk
@r_devops
Why do video hosts still hide pricing behind a "Contact Sales" wall?
I just want to know what 1M video minutes costs before I commit.Platforms like Vimeo and Wistia force you to talk to sales for anything beyond basic tiers. Meanwhile, tools like Kinescope just publish their full pricing publicly.
Why do legacy B2B video tools still do this? Is transparent pricing actually that rare in this space, or is it just a tactic to squeeze more money out of enterprise clients?
https://redd.it/1wjs57e
@r_devops
What is your background?
I have a very unconventional background in the IT world.
After being laid off from my job in the restaurant industry in 2013, I decided to change careers.
I had three options based on my passions: graphic designer, musician, or IT specialist. Since I’d been building my own PCs for a while and had long been passionate about computers (shout-out to manually editing Diablo 2 save files in binary !) I chose to train as a TSRIT (Senior Technician in IT and Telecommunications). I then became a sysadmin and systems integrator at a software company.
I moved up the ranks quickly, becoming a developer for a JavaScript-based launcher, then a DevOps engineer, and now I work as both a DevOps Engineer and a web developer.
It took relentless hard work, morning, noon, and night, but I’m happy today!
What about your career path?
https://redd.it/1wjo6wp
@r_devops
How much do you actually trust your own runbooks during a live incident?
I am researching about on-call engineers problems and finding a fix for it, I have this query:
When you're in mid-incident, do you follow the runbook as written, or treat it as a rough starting point because you've been burned by outdated steps before?
How often do runbooks get updated to reflect what actually happened, versus what was expected to happen?
https://redd.it/1wjhwis
@r_devops
Do you decouple Terraform state within each environment?
For those managing Terraform in production, how do you structure state within each environment?
Do you keep networking, DNS, EKS, database, etc. together, or separate them into independent layers like:
dev/01-networking/
dev/02-dns/
dev/03-eks/
dev/04-database/
with each layer having its own Terraform state?
https://redd.it/1wj76xq
@r_devops
I built Routa an open-source local traffic gateway for developers
I’ve been building Routa, an open-source developer traffic gateway that combines local tunneling, HTTP inspection, request replay, routing, mocking and failure simulation in one tool.
It’s still early, so I’d really appreciate feedback from developers who work with APIs, microservices or webhooks.
GitHub: https://github.com/7uyash/routa/
If you find it useful, a GitHub star would mean a lot ⭐
https://redd.it/1wj2nkn
@r_devops
which of these actually make a retried request safe?
https://redd.it/1wiz6bi
@r_devops
Looking for a docker registery
My company serving 5 apps to the many customers. Managing deployment and new releases are the biggest problem. Because all the time we are copying the build files and pasting to the target server by connecting with rdp. I Want to use dockerfiles for serving and want to keep these images on our server with a registery.
ACR (azure container registery) and the other ones are still the options but i want to host in my server like i did for npm libraries with verdaccio.
And i would like to get your opinnions about my deployment plan. Our customers servers are all windows. We have some windows-only apps. But for the ones which i plan to serve with docker file is better to run with docker. So i plan to use WSL and pull the images from the registery that i am planning to create.
Wdyt?
https://redd.it/1wivbio
@r_devops
Your run-book is still state in someone's head. You find out on the second incident.
Saw the thread here this week about keeping too much system state in your head. Most of the answers were run-books, IaC, docs. I would have given the same answer a year ago and I would have been about half wrong, and it took a second incident to show me which half.
The setup, briefly: automation that writes into the systems of record, tickets and CRM mostly. It proposes a change, a person confirms it, then it writes.
The first incident went the way they all go. Something wrong went in, someone noticed, someone fixed the record by hand, and the retro said we added a check. I have written that sentence in a retro more than once. The check was real, it was in the run-book, and it was also a thing 2 of us remembered.
The second incident is where you find out that a remembered rule is just state in someone's head with better formatting.
What it looked like, all pretty boring:
Someone bulk edits 4 tickets to done inside 2 minutes, and every one of them reads like a legitimate change if you look at it on its own.
A decision arrives from the meeting notes, then the same decision arrives again from the chat thread an hour later, and the second copy opens a second review. Both get worked, by different people.
A CRM stage moves and the person who moved it is not the one who owns the account.
And the check from incident 1 was, on inspection, a rule one person remembered. Written down, sure, but it only really fired when that person was the one looking.
What changed afterwards, and none of it was the model:
The same signal arriving twice does not open a second review any more. When a duplicate gets dropped it gets logged as a decision, so you can go and look at what was dropped and why, instead of discovering later that something quietly vanished.
A review stays open until a person decides it. A lost card or a missed notification does not close one, and 2 people cannot end up working the same row.
Undo goes through the same gate as the original write. Reversing a confirmed change is a new proposed action under the name of whoever approves it, from the ticket or the chat they already have open. Nothing gets deleted to make the problem disappear.
And what gets counted is per source system rather than per person, which sounds like a detail and turned out to be most of the reason people kept using it.
The approval gate is the part everyone asks about and it is honestly the least interesting piece. The interesting part was that incident 2 was boring.
Here is the bit I have not solved, which is why I am posting.
"A review stays open until a person decides it" is the right rule and it is also how you end up with a queue nobody owns. Week 1 everyone works it. Week 3 there are cards sitting open 5 days and the people who could decide them have muted the notifications. Right now a card never expires, by design, so the queue only grows unless somebody works it.
If you run anything with a human decision in the loop, change approval, access requests, deploy sign off, whatever it is: what happens to a card nobody picks up? Does it escalate, does it expire, or does it block the next action until someone deals with it? And who decided that where you work, ops or the business side?
https://redd.it/1wiraia
@r_devops
The integration cost nobody puts in the vendor eval spreadsheet
Every vendor eval spreadsheet has rows for price, features, and support SLAs. None of them have a row for the actual engineering time it takes to get the thing properly integrated and trusted enough to rely on. That cost is real and it's usually bigger than the license fee itself, but nobody puts it in the comparison until it's too late to matter. wanna know how other teams actually account for this before signing, if at all.
https://redd.it/1whvwqm
@r_devops
Do you ever feel like context switching and keeping too much system state in your head is the real bottleneck?
I don’t mean not knowing the tools. More like understanding the pieces individually, but having to keep too many services, dependencies, environments, incidents, or assumptions in mind at once.
If you’ve run into that, what have you actually done about it - anything to improve your ability to hold more context in mind, or mostly systems/tools to reduce how much you need to remember?
https://redd.it/1wi4ec4
@r_devops
Macbook Air vs Macbook Pro
Hello Team,
I work as a Senior DevSecOps Engineer and I have to do a lot of work of Terraform, Kubernetes, AWS, and some security related stuff and documentation.
In your view which Macbook is best to do the above sort of task. I am looking ahead to use this macbook for 6 years down the line.
Need your honest opinion, I know this has been asked a lot of times but still I want some suggestions.
Thanks.
https://redd.it/1whx76m
@r_devops
From DevOps to AI engineer
I still don't understand exactly what is an AI Engineer. But from what I've pieced together, it looks a lot like DevOps but for AI systems: you build, deploy, and maintain AI-powered solutions (integrating LLMs, setting up pipelines, wiring APIs, keeping things running in prod) rather than training models from scratch like an ML/Research Engineer would.
Is that roughly right, or am I oversimplifying? Where's the actual line
https://redd.it/1whii2w
@r_devops
How are teams keeping technical docs in sync with fast release cycles?
We’re a small product and engineering team shipping updates several times a week. As releases have gotten faster keeping our technical docs current has started to become a real problem.
Code changes go through a clear review process, docs are easier to miss. A feature ships, an API changes. Then a few weeks later someone finds an old example or a page that no longer matches the product. For teams with a similar release pace what has worked best for you, im interested to hear what you guys do for smoother workflow on this problem
https://redd.it/1whh2k6
@r_devops
Anyone using MCP to manage Infrastructure
Hi Everyone, I am working in a small company and we do not have the resources to get really big teams. So what we though of was to start using AI to cover the current gap. We mostly suffer from a lot of operational issues which needs to be resolved really fast. (I know really fast is really vague, but sadly there is no SLA because the Manager only sees every hour that our system has issues as lost revenue. its an ecommerce company)
So what we thought of now is to utilize AI and MCPs to help us evaluate issues faster. Like on our File Servers, there are a certain set of conditions that the folders need to have so that our applications work. Sometimes the files that we receive cause issues and we would like to simply utilize AI to evaluate all of this. We are also utilizing AWS and Kubernetes.
Any thoughts on the strategy that we have utilized. I would like to know:
Have you guys done something like this? Any pointers that we need to consider?
Interested on any other innovative ways how you have also solved similar issues?
Other interesting ways that would help us as well and not just using AI and MCP for evaluating issues.
Or just in general thumbs up 😄
Thanks for you feedback!
https://redd.it/1wguwej
@r_devops