r_devops | Unsorted

Telegram-канал r_devops - Reddit DevOps

270

Reddit DevOps. #devops Thanks @reddit2telegram and @r_channels

Subscribe to a channel

Reddit DevOps

How do you catch deploy-unsafe migrations before they hit prod?

We got bitten a couple of times by migrations that were fine as a target schema but not fine during the rollout - old pods still reading a column that a new pod’s migration already dropped. Everything else was set up properly (rolling updates, probes, migration job runs before pods start), didn’t matter.

Until recently our answer was “reviewers should catch it,” which in practice meant sometimes they did.

At Grafana (OnCall team, Django stack) we had django-migration-linter in CI and I honestly forgot how much work it was quietly doing until I no longer had it.

Current stack is Drizzle, no equivalent exists, so we ended up writing our own check: fails the pipeline on drops/renames/NOT-NULL-in-one-step unless the migration is explicitly marked as needing a maintenance window.

Wrote up the rules if anyone wants them: https://archestra.ai/blog/drizzle-migration-linter

For those of you enforcing this in CI, where did you draw the line? Some of these checks (index creation, defaults on big tables) feel like they’d false-positive constantly.

https://redd.it/1u38aug
@r_devops

Читать полностью…

Reddit DevOps

What’s harder in cloud cost optimization: finding savings or getting teams to act?

I’ve been talking to FinOps and DevOps teams lately and noticed an interesting pattern.
Most discussions eventually move away from dashboards and reporting and toward:
Who owns the opportunity?
How do we assign it?
How do we track remediation?
How do we verify the savings actually happened?
Curious about real-world experience:
If you manage cloud costs today, what’s usually the hardest part?
Finding savings opportunities
Prioritizing opportunities
Getting teams to take action
Tracking implementation
Verifying actual savings
Would love to hear where the process breaks down in your organization.

https://redd.it/1u3410l
@r_devops

Читать полностью…

Reddit DevOps

Wrote up how OTel fleet management works under the hood with OpAMP Supervisor
https://telflo.com/blog/fleet-management-effectively-managing-chaos

https://redd.it/1u2y6dw
@r_devops

Читать полностью…

Reddit DevOps

I spent 45 minutes manually failing over LLM traffic at 2am because both Claude and AWS Bedrock hit capacity

I got paged at 2:15am last Tuesday. Claude's API had been throwing 529s for 20 minutes. I had wired up an AWS Bedrock fallback path a month earlier but never tested it in production, so I spent 20 minutes fumbling with config changes and redeploying before traffic started flowing. Then Bedrock started throttling too. By 2:30am both were down or rate limited in our primary region, and I was manually editing env vars to point traffic at a third provider that wasn't even in our fallback chain while trying not to wake up the rest of the team. Forty five minutes later things stabilized. I went back to bed at 3:45am and stared at the ceiling for another hour.

The next morning I realized my retry logic was fine for a single provider dying, but it assumed providers fail independently. They don't. Claude and Bedrock are different APIs, but at the infrastructure layer they're both AWS. Same GPU pools, same regional capacity limits, same quota exhaustion behavior. When the underlying cloud is throttling inference in a region, it doesn't matter which endpoint you're calling.

I knew the market was tight. Hyperscalers are spending $60B plus per quarter on AI infrastructure and still can't keep up. NVIDIA lead times are measured in quarters. Sovereigns are stockpiling chips. But knowing that abstractly and getting paged because of it are very different things.

We started routing through a gateway last week. Looked at LiteLLM proxy first, actually ran it for two days in staging, but the config overhead for multi provider failover was more than I wanted to maintain. Switched to TokenRouter mostly because it handles the latency based provider switching without me writing custom health check logic. It's not magic. We still keep direct API contracts with Claude and Bedrock as backup channels. The gateway is tertiary. If it goes down we fall back to direct calls. Six days in, still validating whether the fallback triggers cleanly under real multi provider outages.

The part I still don't have a good answer for is budget governance. Finance thinks API calls are cheap until they're not. The gateway enforces a max spend per request type, which stopped one runaway job last week. But that's a band aid. The real fix is treating inference as a FinOps discipline, same as compute or storage. Most finance teams aren't there yet.

https://redd.it/1u2socm
@r_devops

Читать полностью…

Reddit DevOps

Find another job or stay current

Im currently a fresh graduate IT admin,but doing devops via ADO (exclusively), basically an IT admin by name only (not doing much IT work).

My question is, shud i stay for like a year, or shud i find another more general IT role like a tech support engineer or it support? Because at some point i do plan on being a cloud engineer. I had one jr. cloud engineer interview before, they said it was a waste for me to quit my current job, as it was a rare opportunity to work in devops from entry lvl.

Would appreciate a no bs answer, if roasting people while giving advice is how u guys like it, im right here 🙏

https://redd.it/1u2ovrs
@r_devops

Читать полностью…

Reddit DevOps

Self-hosted GitHub Actions runners on EKS: the failures that taught me the most
https://redd.it/1u2c4tm
@r_devops

Читать полностью…

Reddit DevOps

Nginx tuning tips: HTTPS/TLS - Turbocharge TTFB/Latency
https://linuxblog.io/nginx-tuning-tips-tls-ssl-https-ttfb-latency/

https://redd.it/1u24ar9
@r_devops

Читать полностью…

Reddit DevOps

Removed by Reddit

Removed by Reddit on account of violating the [content policy. ]

https://redd.it/1u25805
@r_devops

Читать полностью…

Reddit DevOps

How do you share cloud cost findings with non-technical leadership?

In my experience, DevOps teams often identify waste in AWS/Azure/GCP, but the challenge is communicating it to CFOs and executives.

Do you export reports from Cost Explorer?

Use dashboards?

Build custom reports?

What’s your current workflow?

https://redd.it/1u223gt
@r_devops

Читать полностью…

Reddit DevOps

Do static inventories alone create false positives and remediation noise?

A lot of the container security discussion lately seems to revolve around CVE counts and vulnerability scans. But the more I look into it, the more I wonder how much of the noise comes from relying on static inventories alone. SBOMs and package inventories tell us what's present in an image, but they don't necessarily tell us what's actually used at runtime. As a result, security teams can end up spending time investigating vulnerabilities in components that may never execute in production.

Alpine or distroless images are small, but that doesn’t automatically mean they’re hardened. Some hardened images still carry packages that never get used at runtime. And a lot of standard base images inherit hundreds of vulnerabilities simply because they include way more software than the workload actually needs. The operational problem is that all of this eventually turns into scanner noise, triage fatigue, delayed approvals, and teams shipping with exceptions because fixing everything isn’t realistic.

Lately I’ve been seeing more discussion around curated near zero CVE images and runtime informed hardening as a way to reduce risk earlier in CI/CD instead of just managing huge CVE backlogs later. How are people thinking about this now? Are teams actively trying to reduce what’s inside images, or mostly relying on scanning and patching after the fact? I’m puzzled.

https://redd.it/1u1ue7u
@r_devops

Читать полностью…

Reddit DevOps

What are some of the most impressive tasks you’ve automated?

Title

https://redd.it/1u1se9v
@r_devops

Читать полностью…

Reddit DevOps

Built this dev tool to help fine tune vLLM server

It measures live GPU and scheduler telemetry against your hardware’s real decode/prefill ceiling.

Traffic-Gated Analysis: Idle servers are ignored. Under load, it ranks five production bottlenecks (under-batching, KV pressure, prefix reuse, concurrency saturation, OOM risk) and surfaces exactly one prescriptive fix per iteration.

Closed-Loop Verification: Run profile diagnose, apply the suggested fix, and Profile automatically detects the vLLM restart. It re-measures and reports throughput, latency, efficiency, and cost deltas with a definitive better/worse/plateau signal.

Zero Overhead: Shipped as a single Rust binary. Scrape-only (NVML + /metrics), strictly local, with zero external APIs.

The output tells you what happened, why it was slow, and exactly what to fix first.

Profile: https://github.com/jungledesh/profile
Docs: https://jungledesh.github.io/profile/index.html

If you are running vLLM in production, give it a try:
Try it: profile diagnose --duration 2m

https://redd.it/1u1ds8c
@r_devops

Читать полностью…

Reddit DevOps

Code Beam Europe 2026 Early Bird tickets dropping soon
https://redd.it/1u1esup
@r_devops

Читать полностью…

Reddit DevOps

What are some tasks in daily DevOps life that you think agents based on frontier models (like Opus 4.8) can't solve?

I've been playing around with agents, trying to work on DevOps management tasks (including provisioning, updating, monitoring and troubleshooting across different parts of the infra lifecycle, starting from Terraform/Pulumi to Kubernetes, Prometheus, and app layer tools), and wanted some feedback on what tasks in your usage of these tools you think frontier models do not really achieve what they're set out to do.
I've discovered certain patterns, but they're pretty niche in general. I'm looking for feedback on something more realistic. Curious if folks out here have struggled with some tasks using the frontier models in general.

Disclosure: I'm an academic looking at DevOps automation in general, using agents, and want some community feedback to ground my work. Thanks!

https://redd.it/1u1cm2c
@r_devops

Читать полностью…

Reddit DevOps

When the only person who knew how to do something left or went on holiday — what actually happened?

I’m an engineer trying to understand how teams handle critical knowledge that lives in one person’s head and never gets written down such as deployment steps, a gnarly config process etc

Genuinely curious about real stories, not theory:

\- What was the knowledge, and what happened when that person wasn’t around?

\- How long did recovery or handover actually take? Did anything break?

\- Did your team ever fix it properly afterwards, or just hope it doesn’t happen again?

https://redd.it/1u18fro
@r_devops

Читать полностью…

Reddit DevOps

Are any of the AI tools actually worth learning?

Hi. I'm currently only using claude or copilot to read my code / infra project, prompt it to add something there or, give it some error message to analyze. But on youtube or other places I'm always seeing these videos people talking about loops, agent, "automated ai-based ​troubleshooting",... .

Is any of this actually worth digging into? Or its all just hype? Especially now since the token usage has become limited in most companies.

https://redd.it/1u3670t
@r_devops

Читать полностью…

Reddit DevOps

I truly don't see the point.
https://redd.it/1u2zk63
@r_devops

Читать полностью…

Reddit DevOps

Fun project to use on GCP VM

Hey! I’m learning GCP and planning deploy my first VM. I’d love to use it for a fun small project while I learn, any ideas or suggestions?

https://redd.it/1u2sd09
@r_devops

Читать полностью…

Reddit DevOps

Open source tools are the DIY of the software world

I was just looking at how easy plug and play some open source solutions are. And some of these tools are so deeply embedded in everyday infrastructure that you don't even register them as open source anymore. For examples Mozilla for web, Kubernetes for production workloads, Linux the most popular OS System. These are not hobby projects that got lucky. They became foundational precisely because they were genuinely open.

It is debatable here that there is a tradeoff between the extent of openness and keeping revenue leakage in check. You cannot run a company on goodwill alone. Engineers need salaries, infrastructure costs real money, and a sustainable open source project usually needs a commercial entity behind it to survive past the first wave of enthusiasm.

What does belong behind a paywall is the scale and operational story. Multi-tenant management for organizations running hundreds of instances. SLA-backed support. Compliance certifications that require ongoing audit work. Advanced analytics that only matter once you have a team large enough to need them. Managed hosting for teams that do not want to run the tool themselves. These are real costs, and customers who need them generally understand that they should pay for them. The line is not arbitrary. It is the line between what every user needs to be productive and what only some users need to be at enterprise scale.

For better user friendliness that you can make your product in phases and let the customer use your product for free. By the time their company grows to the point where they need the paid tier you are selling them an upgrade on something they already trust.

https://redd.it/1u2nykw
@r_devops

Читать полностью…

Reddit DevOps

useful tools for cleaning up messy infra / cloud costs

putting together a small list of tools that are actually useful when you’re dealing with messy infra, noisy cloud bills, random k8s waste, and storage stuff that nobody wants to touch.

not a “best tools ever” list, just things that seem useful depending on the problem.

kubecost
good if you’re running kubernetes and need to understand where spend is going. especially useful for finding oversized workloads, unused resources, namespace/team-level waste, and pvc cost creep.

vantage
better for general cloud cost visibility across AWS. nice if you want a cleaner view of spend, trends, unused resources, and the usual “why did the bill jump?” type questions.

cloudhealth
more enterprise-y, but useful in bigger orgs where finance, infra, and leadership all need reporting. not really a fix-it tool, more of a visibility/governance tool.

datafy
interesting for the storage side specifically. most cost tools can tell you that EBS volumes are overprovisioned, but they don’t help much with reclaiming that space. Datafy seems more focused on EBS storage optimization/reclamation instead of just another dashboard.

netdata
good for quick host-level visibility. useful when you just want to see what’s happening on a machine without setting up a huge observability stack.

restic
solid backup tool. simple, boring, reliable. still one of those tools that makes sense when you want backups without too much drama.

btop
not really a cloud cost tool, but still useful for quick server checks. sometimes you just need to ssh in and see what’s going on.

curious what else people are using for infra cleanup, storage waste, and cloud cost problems that actually helps beyond just making another dashboard.

https://redd.it/1u2enyv
@r_devops

Читать полностью…

Reddit DevOps

eBPF based evals have just been amazing
https://www.emphere.com/blog/testing-a-security-tool

https://redd.it/1u27wis
@r_devops

Читать полностью…

Reddit DevOps

First job in devops. What should I focus on?

i just got my first job as jr devops engineer(2nd week) in a really nice company, before this i was in startups as (shopify+wordpress+IT) first time going in dedicated role, manager asked me to build pipelines for open source projects which i did pretty much easily. this company uses both windows and linux servers (on-prem and cloud as well. what do you guys recommend should i focus on in terms of excelling in this company and career keeping in mind that this is my first devops role and I've done little self learning. i know i can just google this stuff but talking to real person and get their point of view felt nice so pls be lenient if you find any question foolish.

https://redd.it/1u24y4t
@r_devops

Читать полностью…

Reddit DevOps

Junior devops community india

Can anyone add me to any fresher devops community from india

Just starting devops and wants to be in a community

https://redd.it/1u21o85
@r_devops

Читать полностью…

Reddit DevOps

Hermetic Build using Maven

I want to implement a hermetic build for a pseudo supply chain that im Tinkering on.

1st of all is it even possible to do a hermetic build using maven or is bazel the only choice ?

And what about the lockfile plugin by ChainsProject, how far can you stretch it

How do peeps who actually work devops implement Hermetic build using maven?

https://redd.it/1u1vcl2
@r_devops

Читать полностью…

Reddit DevOps

Are AI agents reintroducing problems software engineering already solved?

Working with agent workflows lately, I've started feeling like we're just reintroducing a bunch of problems software engineering already spent years solving. Once an agent gets past the "Hello World" stage, its behavior depends on a mix of prompts, tool permissions, memory, retrieval settings, and whatever model endpoint happens to be up. A lot of that state is runtime-driven or buried inside framework abstractions. Trying to reliably review, reproduce, or audit it becomes much harder compared to the static code workflows most of us are used to.

We've spent decades building mature workflows around version control, CI/CD, PR reviews, rollback capability, and environment separation so you actually know what binary is running in prod and what changed since the last incident. With agents, a lot of behavior still seems to be assembled dynamically at runtime instead of being treated as a properly versioned artifact.

How are teams actually handling this in production? Are people moving toward declarative, git-based definitions for agent workflows, or is the ecosystem still too fragmented and framework-specific for that to work cleanly? GitHub Next shipped Agentic Workflows, gitagent exists, and Claude Code already leans heavily into git-native workflows. The direction clearly has traction now, even if the ecosystem hasn't converged yet.

https://redd.it/1u1u7rp
@r_devops

Читать полностью…

Reddit DevOps

Pivot to Devops from infra guy

Hey everyone,
I am currently looking at a career pivot from a generalist / infra / sysadmin guy to DevOps. 30 YO male, EU, 10 years in IT without college degree, 6 of those years are in a sysadmin role.

In my current position, I manage some onprem / azure servers, dabble in networking, and do a lot pf scripting in powershell to automate a lot of things. I would not really call myself too skilled at programming though. I would overall consider myself medior to senior in this role.

I understand more or less what DevOps entails, but i do not know where to start exactly. My org is not really into modernizing things, so I do not have any experience with containers or ci/cd, everything is still running on VMs. I do try to actively upskill though in my own time.

Now my question is, where to start?
Containers / kubernetes / docker
\- I am currently playing with this in my homelab, still very green though.

Ci/CD
\- dont even know where to start on this one

Git
\- playing with this in my current org. Pushed all my pwsh scripts to an Azure DevOps and playing around with it. Still have some holes here.

Python
\- Do I absolutely need this one? I guess I can read it, therefore I can vibe code and check if the Ai code is not an absolute mess, but again, I do not consider myself very strong programmer and I would struggle with this the most.

IaC
\- playing around with this in my org azure environment. I pushed a few server with biceps and terraform, but I do not really create servers that often to make use of it that much. Seems straightforward enough though.

What would you focus on if you were in my shoes? How long do you think learning all this can take me to make the pivot? Will be happy for all advice.

https://redd.it/1u1svq6
@r_devops

Читать полностью…

Reddit DevOps

Increased Fargate Spot interruptions in ap-south-1 (Mumbai) — anyone else?

Over the past week or so my Fargate Spot tasks are getting interrupted way more often in ap-south-1 than they used to. It was rare before, now it's near-constant. Nothing on the public status page and nothing in my personal Health Dashboard.

Is anyone else running Fargate in Mumbai seeing this right now, or is it just my account/AZ? Mainly trying to figure out if it's a transient capacity crunch or the new normal.

https://redd.it/1u1e916
@r_devops

Читать полностью…

Reddit DevOps

prompt drift is an operations nightmare. we started using gitops for our agents.

We’ve been running a few LLM agents in production and managing them has been a complete mess from an operations perspective.

Right now, most teams seem to store agent prompts, system instructions and tool schemas in application databases or behind some vendor's UI dashboard. The problem is we have zero visibility into configuration drift. If an agent starts acting up or throwing weird tool execution errors, it's impossible to track the root cause. Did an upstream model change? Did a PM silently tweak a prompt template in a DB? Who approved that new DB tool access? There’s no audit trail, no commit history, and zero pipeline validation.

An AI agent makes active runtime decisions. It needs to be treated like infrastructure code (Kubernetes manifests or Terraform), not application data.

To fix this, we started forcing our agents into a standard GitOps workflow using an open-source spec called Lyzr GitAgent.

Instead of configuring things in a UI or database, everything is just flat files in a repo:

agent.yaml for the core runtime config and model parameters.

`soul.md` for the raw system prompts and behavioral guardrails.

/skills directory where python/bash tools live as testable scripts.

The upside is we can now drop validation commands directly into our standard CI/CD pipelines to lint configs before deployment. If someone wants to update an agent’s behavior, they open a normal GitHub PR. It gets peer-reviewed, goes through a human approval gate and deploys as an immutable commit. If things go sideways in production, we just git revert.

How are the rest of you handling compliance and tracking prompt drift for enterprise agents?

https://redd.it/1u1el4c
@r_devops

Читать полностью…

Reddit DevOps

That "CVSS 10.0" Azure DevOps CVE (2026-42826) is real, already fixed by Microsoft, and a useful lesson in not panicking at a score

Saw a few people spinning up about CVE-2026-42826 since it's carrying a perfect 10.0, so a quick PSA + a point worth discussing.

What it is: a critical information disclosure flaw in Azure DevOps (CWE-200) that allowed unauthenticated remote attackers to disclose sensitive information over the network CVSS 10.0. On paper that's about as bad as it gets: no auth, no user interaction, network vector. CrowdStrike

The part that matters for us: there is nothing to do. Microsoft remediated it entirely in their cloud infrastructure and explicitly stated there's no action for users of the service to take the CVE exists mainly for transparency. If you're on the hosted Azure DevOps Services, you were patched before you ever heard about it. There's no public PoC and the exploit maturity is rated unproven. The RegisterTheHackerWire

The reason I think it's worth a thread anyway: it's a clean example of why a raw CVSS base score is a terrible prioritization tool on its own. A 10.0 that's already mitigated cloud-side with no PoC is operationally a non-event, while a 6.5 in a transitive dep that's actually reachable in your prod path could ruin your week. The base score doesn't carry that context the temporal and environmental metrics (and honestly, just reading the advisory) do.

It's also a reminder of why Azure DevOps is such a juicy target regardless of this specific bug: these environments routinely hold deployment credentials, cloud secrets, CI/CD tokens, infra configs, and source code. An info-disclosure flaw there isn't theoretical it's a direct line to everything downstream. INE

https://redd.it/1u19lv7
@r_devops

Читать полностью…

Reddit DevOps

It's a good idea for improve my cloud knowledge?

Estou trabalhando em um projeto onde minhas responsabilidades são múltiplas: DevOps, SysAdmin, QA/Testador. Basicamente, eu apoio uma equipe de desenvolvimento com tarefas de atribuições de DevOps, como: criar e manter helm-charts para implantar microsserviços no ArgoCD, configurar e manter o fluxo de CI/CD do GitLab e trabalhar com boas práticas de GitOps (usando branches semânticas e jobs). Mas, esse projeto está quase no fim. Tenho procurado outras oportunidades de emprego semelhantes, mas meu pouco conhecimento em gerenciamento de nuvem tem me barrado na maioria das aplicações de emprego. Por causa disso, agora estou voltando aos meus estudos para renovar minha certificação AWS Certified Cloud Practitioner (estou fazendo o AWS Cloud Quest: Recertify Cloud Practitioner). Mas eu acho que isso não é suficiente para me candidatar a oportunidades internacionais de engenharia DevOps.

Vale a pena construir um serviço público na web (um exemplo, um site), com arquitetura em Kubernetes usando instâncias EKS? Eu sei, talvez eu precise gastar um pouco de dinheiro para isso. Mas minhas intenções são demonstrar meu conhecimento com projetos reais. O que você acha disso? É um bom plano?

Sobre K8s, passei por algumas situações práticas, que melhoraram meu conhecimento, como remoção acidental de namespace, migração do nginx-ingress para o gateway envoy, configuração e uso de segredos seguros com segredos externos, etc. Eu passei no exame KCNA há um ano.

https://redd.it/1u17ly3
@r_devops

Читать полностью…
Subscribe to a channel