270
Reddit DevOps. #devops Thanks @reddit2telegram and @r_channels
Any suggestions on DevOps Certifications for better opportunities?
Pretty much what the title says. Looking to pivot/advance my career and wondering which DevOps certifications are actually worth the grind right now.
AWS? CKA? Something else? Would love to hear what helped you land interviews or a bump in pay.
https://redd.it/1xnhklc
@r_devops
Teams are shipping LLM features from random pods. How do you even inventory which services call which model APIs?
Platform lead at a \~150 engineer SaaS company, everything on EKS. In the last 6 months at least 5 product teams have shipped something that calls OpenAI, anthropic or bedrock, plus one team self hosting an open model on a GPU node pool.
Security asked me a simple question last week, "which services send customer data to which model prociders?" I couldt answer it. API keys live in a mix of secrets manager, k8s secrets and in one case an env var in a Helm values file. Egress is mostly open because locking it down broke too many things 2 years ago.
Things i've considered: -Egress proxy for all outbound traffic to known ai endpoints. Probably the right long term answer, but its a big change and the teams will hate it. -Grepping repos for SDK imports. Fast, but misses anything calling the APIs over plain HTTP, and says nothing about what's actually running. -Our security team is evaluating upwind, and in the demo is showed outbound calls from workloads to ai providers at runtime, grouped by service. Im curious whether that's useful in practice or just a nice demo screen.
How are other platform teams handling this? Did you go proxy first, or start with visibility and lock down later?
https://redd.it/1xndmow
@r_devops
Is Google Cloud Kubernetes Engine built-in Observability enough?
Hello. I just started being responsible of a Kubernetes cluster. I have 0 experience on Kubernetes, besides having deployed the GKE cluster successfully (which honestly is quite straightforward) in Google Cloud.
The cluster has only one node pool, with 1 helm chart and a couple of other manifests. Most likely it's complexity won't go up and the scalability needed in terms of pods/nodes is not very high.
Is the built-in observability in GKE enough to monitor/observe a cluster? Or would it be good to have something like Prometheus + Grafana? Also, any tips or links with advices about what to keep an eye on?
Thank you very much
https://redd.it/1xmwkvn
@r_devops
Am I wrong to be turned off by coding examinations in interviews?
I've been speaking to recruiters recently about roles in my area.
Almost every set of interviews now has a live coding portion to it.
I've got an interview next week where I've been told I will be asked to live code. When asked what sort of coding, I was told - it could be writing Terraform, Python, writing a helm chart, kubernetes manifests. It will be random on the day.
I've individually done a lot of these. I've been writing Terraform for years, and I do Kubernetes manifests every single day.
However, I might not touch Terraform for months whilst I work on one project. Python might be even longer.
If tasked with a Terraform project, I'll look at some recent work I did, get a feel for it again, and I'm productive pretty quickly.
But just throw me into an interview, and have multiple engineers ask me to live code, and I feel like I would turn to jelly. I can't remember off the top of my head how to do a module for Terraform - give me a previous one I created, I can take it as a template and start rewriting it, looking up functions if I need new things.
Same for Kubernetes. I will take a deployment file and a svc I did for something in the past, copy and paste it into a new file and edit it to fit my new needs. Writing the document from scratch without documentation... I can't remember that for shit. Who does it afresh on every new service?
https://redd.it/1xmsxbb
@r_devops
GitHub Actions now supports dependency lockfiles
We often talk about securing the software supply chain in terms of application dependencies. For instance, pinning package versions, maintaining lockfiles, and auditing third-party libraries.
But what about the code that runs our CI/CD pipelines?
GitHub Actions workflows can execute third-party actions, and the security of those dependencies matters just as much when building and deploying software.
The introduction of workflow dependency lockfiles is an interesting step towards making CI/CD workflows more reproducible and secure.
I wrote about this recently, exploring why workflow lockfiles matter and how they fit into the broader software supply-chain security picture.
Read the article: https://amitmerchant.com/ci-pipeline-lockfile/
Curious how others are approaching dependency security in their CI/CD pipelines. Are you already pinning your actions and enforcing similar controls?
https://redd.it/1xmlv6c
@r_devops
I built PagerWho, an open-source, self-hosted incident management and on-call tool (MIT)
Hey r/devops,
I've been building PagerWho, an open-source incident management and on-call platform you can self-host with Docker Compose.
I started it because small and mid-size teams often just need the basics: alerts coming in, the right person getting paged, and escalation if nobody responds. Doing that well shouldn't require per-seat pricing or sending your incident data to another SaaS.
How it works
1. Your monitoring tool sends alerts to a service's webhook. Generic JSON, Grafana, Sentry, and Datadog formats are supported. Duplicate alerts are grouped into one incident.
2. The service's escalation policy decides who gets paged. Steps can target a user, a team, or whoever is primary or secondary on call.
3. Responders are notified via Slack, email, SMS, or phone call. If nobody acknowledges in time, the incident escalates to the next step and can repeat.
4. Everything is recorded on the incident timeline: when it fired, who was notified, delivery results, acknowledgement, and resolution.
Features
\- Services with owners and per-service webhooks
\- Primary and secondary on-call rotations, with schedule previews and overrides
\- Escalation policies with delays and repeat rules
\- Notification delivery tracking with retries and dead letters, so you can see why a page didn't go out
\- Event-driven workflow automation
\- Project roles and permissions, audit logs, and operational diagnostics
\- Optional AI assistant using yy default)
Ops-friendly by design
\- The only dependency is PostgreSQL: no Redis or message queue. The background worker uses
DB claims and leases, so you
\- Docker Compose for local evaluation and for production
\- OpenAPI docs served at /api-docs
\- MIT licensed
Try it
git clone https://github.com/Pager-Who/PagerWho
cd PagerWho
docker compose up --build -d --wait
GitHub: https://github.com/Pager-Who/PagerWho
Hosted version, if you'd rather not run it yourself: https://pagerwho.com
It's still early, so I'd really appreciate feedback. What would stop you from using this
instead of PagerDuty or Opsgeniures matter most to you?
Thanks for reading!
https://preview.redd.it/62oiyl68obzh1.png?width=1860&format=png&auto=webp&s=26936899ecf1480860c85a25310f34ad92417d1d
https://redd.it/1xmj3ey
@r_devops
OpenObserve vs Stack Grafana
Hey everyone, how are you? I'm seriously thinking about setting up an OpenObserve POC to get a better understanding of how it works.
Has anyone been using it?
https://redd.it/1xm3ts3
@r_devops
What does one mean by "projects" if not homelabs? Are homelabs considered as projects?
Hey guys,
Help me understand what does one mean by "projects".In DevOps you homelabs are your projects right? If anyone of you had developed any open source DevOps projects I'd like to have a look at it .
https://redd.it/1xly6f1
@r_devops
Unexpected cloud cost spike. How to handle postmortem?
We had a recent incident where an auto-scaling config kept launching additional instances. No one noticed at first, and it happened long enough that our cloud services bill spiked pretty dramatically before somebody spotted the issue. I'm dreading having to explain this to higher ups, which is happening next week. We've already done a postmortem, but I want to be really prepared when I talk about this.
How have you handled similar situations? What did you change moving forward, especially related to identifying root causes and setting limits on resource scaling?
https://redd.it/1xlvd19
@r_devops
My agent push warning arrived after the push had already left
Claude Code runs in repos where other sessions are working too, so a git push can carry commits that session never made. In October i added a warning that lists those commits. The warning came back with the tool result, and by then the push was done.
So it runs before the command now. Each commit a session makes goes into a small ledger, and on push:
outgoing=$(git rev-list "$ref" --not --remotes="$remote")
foreign=$(comm -23 <(sort <<<"$outgoing") <(sort "$ledger"))
[ -n "$foreign" \] && deny "this push carries commits you didn't make"
The agent can't override it alone. The override only goes in after i say ok in the chat.
Of 78 hard denies in the last 30 days, 58 were this rule.
Anyone else gating agent pushes before they run, or is branch protection enough for you?
https://redd.it/1xltnnz
@r_devops
What’s your experience using incident.io for on-call and incident response?
Hey folks, I’ve been looking into incident.io and trying to understand how teams use it in practice.
What does your workflow usually look like from getting paged to resolving an incident? What does it handle well, and what still needs a lot of manual work or switching between tools?
Also curious about the AI side. If you’ve tried its AI features, have they actually helped during real incidents? If you could improve one part of your workflow with AI, what would it be?
Would appreciate any real-world experiences. Still learning how all these pieces fit together.
https://redd.it/1xlom45
@r_devops
AI taking over Devops.
Hello. i am Starting to learn IT and i still have time to choose my carreer choice. is it wise to try and learn devops or cloud engineering because of AI? im scared that by the time im done with my degree these jobs wont exist.
https://redd.it/1xldkq0
@r_devops
Guys interviewer asked me "why don't you have any certification" and you all say that it's not important.
https://redd.it/1xkzgkm
@r_devops
Building a fast perimeter security auditor in Next.js: Handling Anti-SSRF, RFC 1918 ranges, and socket timeouts
Processing img zovm095m2wyh1...
**Disclaimer:** Personal affiliation — I am the author of this free, open-source project.
Hey everyone,
I built an automated perimeter security auditing tool using Next.js (App Router), TypeScript, and Node.js network sockets.
Many cloud audit tools charge high subscription fees just to check a few HTTP headers, negotiate TLS, and verify basic socket exposure. I wanted to build a fast, deterministic alternative and deploy it live.
When building an external auditing service, the primary security challenge is preventing malicious users from turning the backend into an open proxy or an internal network scanner (SSRF). Here is how I approached the hardening:
1. **Strict Anti-SSRF Protection:** Filtering and blocking all RFC 1918 private ranges (`10.0.0.0/8`, `172.16.0.0/12`, `192.168.0.0/16`), loopback (`127.0.0.1`), and link-local addresses before attempting any socket connection.
2. **Redirect Bypass Defense:** Enforcing `redirect: 'manual'` on all HTTP transport checks so the scanner backend never silently follows redirects into internal endpoints or cloud metadata services.
3. **Slowloris & Hanging Sockets:** Implementing strict abort signals using `AbortSignal.timeout(4000)` along with defensive socket timeouts to keep serverless execution runs clean and responsive.
4. **OWASP Compliance & Fallback Engine:** Evaluating critical headers (HSTS, CSP, X-Frame-Options, X-Content-Type-Options, Referrer-Policy) with a deterministic local engine that guarantees report generation even during API outages.
The platform also generates a clean, print-ready A4 forensic PDF report directly in the browser.
* **Live Demo:** https://security-score-seven.vercel.app
* **GitHub Repository:** https://github.com/martincastillo1996/security-score
IN:https://www.linkedin.com/in/karol-martin-castillo-rojas-17a496168/
I would love to get your feedback on edge cases, socket behavior, or other perimeter vectors worth adding.
**Nota de transparencia:** Proyecto personal de código abierto, gratuito y sin fines comerciales.
Hola a todos,
Desarrollé una herramienta automatizada de auditoría perimetral construida con Next.js (App Router), TypeScript y sockets de red en Node.js.
Muchas plataformas cobran suscripciones costosas únicamente por validar un certificado TLS, revisar cabeceras de transporte y verificar el estado básico de puertos. El objetivo fue diseñar un motor rápido, determinista y desplegado en producción.
Al construir un servicio de auditoría externa, el principal riesgo técnico es evitar que atacantes utilicen el backend como proxy abierto o vector de ataque contra redes internas (SSRF). Así resolví el blindaje:
1. **Protección Anti-SSRF estricta:** Bloqueo de rangos privados según RFC 1918 (`10.0.0.0/8`, `172.16.0.0/12`, `192.168.0.0/16`), interfaces locales (`127.0.0.1`) y direcciones link-local antes de iniciar cualquier conexión TCP.
2. **Control de bypass por redirecciones:** Configuración obligatoria de `redirect: 'manual'` en las consultas de transporte HTTP, evitando que el escáner siga redirecciones hacia servicios internos o metadatos de nube.
3. **Prevención de sockets congelados:** Integración de `AbortSignal.timeout(4000)` y cierres forzados por tiempo de espera defensivo para evitar bloqueos en ejecuciones serverless.
4. **Cumplimiento OWASP y motor de respaldo:** Evaluación de cabeceras esenciales (HSTS, CSP, X-Frame-Options, X-Content-Type-Options, Referrer-Policy) mediante reglas locales deterministas, garantizando parches de remediación aun sin conexión externa.
El sistema además genera un dictamen forense en formato PDF A4 listo para impresión técnica directamente desde el navegador.
* **Demo en vivo:**
How do you clean up stale branches, old PRs, and dead repos across a whole GitHub org?
I manage a lot of repos, that are changed by multiple devs and every few months it turns into a mess: hundreds of merged branches nobody deleted, PRs that went quiet a year ago, repos nobody has touched since a project ended. GitHub only lets me deal with it one repo at a time, so mostly I just... don't.
Curious how others handle this:
\- Do you run a script (gh CLI, an Action), use a tool, or just ignore it?
\- What's the most annoying part?
\- What would you never trust a tool to do automatically?
Especially interested in hearing from people managing 30+ repos or a team org.
https://redd.it/1xk62rx
@r_devops
Railway's headline uptime went from 97.10% to 100%.
Just went back to Railway's status page after the Oct 21–23 outage. The main “Railway” uptime chart now says 100.00% for the last 90 days.
Couldn't access the dashboard, couldn't build anything. Apparently that's 100% uptime. Their own incident updates said builds were paused across every region, and Wayback saved the outage announcement.
The October 30 snapshot shows that same “Railway” chart at 97.10%. Now it's 100.00%. The outage is still inside both 90-day windows.
Support & Community used to sit inside that group. Now it's listed separately, still at 97.10%, while “Railway” gets to show a perfect 100%. Very convenient.
The dashboard and API charts were already showing 100% in the old snapshot, despite their own updates saying those services were unavailable. So those numbers never reflected the outage either.
Yes, the incident is still under History. But if I have to dig through incident logs to find the downtime behind your 100% uptime chart, what the hell is the chart for?
https://redd.it/1xnfgkx
@r_devops
When the dashboard is green but users say it is broken, where do you look first?
This has happened to us more than once. Every check we owned said the system was healthy, response times were fine and no errors were showing. Users were still stuck, usually on one path we did not watch closely, an old client, a single region or a step that depended on someone else.
We started keeping one simple user journey checked from the outside, written in plain words a non specialist could follow when it failed. I am curious what others check first when the green lights and the user reports disagree.
https://redd.it/1xn6zl5
@r_devops
With 4 engineers, do you page 24/7 or business hours plus SEV1 at night?
We're setting this up again for a team of 4, and it's the same argument every time. How we do it today: weekly primary and secondary, handover on Wednesday at 10:00, and below 4 or 5 people we keep nights to SEV1 only, the rest waits for the morning.
Not everyone on the team agrees. Some of us would rather page 24/7 from day one with a stipend, because "SEV1 only" means somebody decides what SEV1 is at 3am, and the severity table had better exist before that call.
For the people who page 24/7 with 3 or 4 engineers, how long did it hold before someone burned out or the rotation changed?
https://redd.it/1xmufk3
@r_devops
How much linux for cloud and devops?
Hey everyone , i am currently learning linux and my ultimate goal is to get a cloud role job. I am confused about how much linux is required for me to learn, which concepts should i focus on that are actually usefull in real world. If anyone is working as a cloud engineer or another cloud role please help ..
https://redd.it/1xmqc02
@r_devops
Which YouTube channels actually help with platform/infra work?
I wish to trim my subscriptions to ones worth thewatching time . Who do you watch for K8s, internal platforms, and lately LLM infra (gateways, routing, cost/observability)?
Deep-dives over hype preferred.
https://redd.it/1xmjgwk
@r_devops
Your AI agent passed CI. But do you know what it actually did?
I've been thinking about a problem with AI agents running in CI/CD.
We spend a lot of time defining what a pipeline should execute, but once we introduce AI agents, things get less predictable.
An agent might pass every test while exercising capabilities we never explicitly reviewed.
So I started building Drifti, an open-source CLI written in Rust.
The idea is to make an agent's exercised capabilities explicit and reviewable.
The workflow looks like this:
1. Observe what an agent actually does.
2. Learn its filesystem, process, and network capabilities.
3. Generate a proposed drifti.yaml contract.
4. Review and commit that contract.
5. Verify future executions against it.
If an execution violates the contract, verification returns exit code 121, which CI can use to fail the job.
One design decision I'm particularly interested in: incomplete observation must never be treated as success. If Drifti doesn't have enough evidence, it returns an indeterminate result (122).
The current MVP uses Linux ptrace, stores evidence locally in SQLite, and keeps policy evaluation deterministic.
It's not a sandbox, and the observation coverage still has limitations.
I'm sharing it early because I'd rather get feedback from engineers dealing with these problems than build something nobody needs.
GitHub: https://github.com/lucabecci/drifti
For those running agents in CI: are you explicitly tracking their capabilities, or are you mostly relying on runner/container permissions?
https://redd.it/1xmgw5g
@r_devops
Watch this story by sudo learn on Instagram before it disappears.
https://www.instagram.com/stories/u_do_learn/4021698308821210323?utm_source%3Dig_story_item_share%26psln%3DMWVteWx3MmptejEyZA%3D%3D
https://redd.it/1xm54fj
@r_devops
Like to improve my networking knowledge
Hello fellow devops engineers,
I would like to improve my networking knowledge do you have some suggestion for any udemy or youtube course to become better at networking.
I am not a noob but i feel i am not good with networking. Some suggestion would be helpful
https://redd.it/1xlw1rn
@r_devops
Portfolio review
Review my portfolio
https://Sumit.cloud
https://redd.it/1xlthv2
@r_devops
Weekly Self Promotion Thread
Hey r/devops, welcome to our weekly self-promotion thread!
Feel free to use this thread to promote any projects, ideas, or any repos you're wanting to share. Please keep in mind that we ask you to stay friendly, civil, and adhere to the subreddit rules!
https://redd.it/1xlqwcb
@r_devops
How many apps have non-engineers at your company built with Claude/Cursor/Lovable, and who owns them when they break?
I’m an ML engineer. Last month I got a Slack from marketing asking why “their dashboard” stopped updating. Nobody on my team knew it existed. It was a Lovable app hitting our Supabase with a service key, deployed to Vercel under someone’s personal account, and the person who built it had left.
That got me wondering how common this is. So, for people on platform/eng/IT teams:
* Roughly how many of these are running in your company right now? Could you actually list them?
* When one breaks, who finds out, and who fixes it?
* Have you tried to get a handle on it (inventory, policy, blocking, importing into Retool/Superblocks), and what happened?
Not selling anything. Trying to understand whether this is a real problem or just my company
https://redd.it/1xlhrh8
@r_devops
Anyone interviewed for SRE at SpaceX?
Hey everyone, I have a technical interview coming up for a Site Reliability Engineer (Data Center Infrastructure) position at SpaceX. It's a non-senior role on the Application Software team.
I was wondering if anyone here has gone through the interview process for this position or something similar at SpaceX. I'm mainly curious about what kind of technical questions to expect and whether there's typically a coding portion. If there is coding, is it more practical scripting or algorithm-style questions?
I'd appreciate any advice or insight from anyone who's been through it!
https://redd.it/1xl1ljl
@r_devops
https://security-score-seven.vercel.app
* **Repositorio de GitHub:** https://github.com/martincastillo1996/security-score
Me gustaría recibir su feedback técnico: ¿qué otros vectores de red o validaciones perimetrales consideran indispensables evaluar?
https://redd.it/1xkomdn
@r_devops
Anyone rethinking IDE extensions after the Cursor marketplace incident?
Was reading about the malicious Claude code extension that show up in Cusor marketplace. The pary that got me was it trying to add an SSH key and open a reverse tunnel l, from something that just look like another editor extension
We talk a lot abt checking npm package and dependencies, but IDE extension feal like one of those thing people install once and forget abt it. Especially with auto updates enbled
Those things make me wonder how much access we are giving these extension without really thinking abt it. Are people actually restricting extension on work machines or is it mostly up to each dev? Curious if anyone team changed anything after this 🤔
Source:https://cursor.com/blog/claude-code-extension-security-notic
https://redd.it/1xkexud
@r_devops
Which is more used in industry: ArgoCD or Flux?
I'm about three-quarters of the way migrating my application infrastructure from ArgoCD to Flux, and man, is it difficult. I had to use both Claude and ChatGPT pretty extensively to parse the absolute mountain of error logs I was getting and correct the pile of bugs that I had inadvertently introduced while setting it up the first time around.
I'm too far into this process to back out, but just in case I finish this process and then realize that I actually didn't want to migrate, which platform is actually used more commonly in the industry, so I should actually spend more energy learning it?
Note, even though I do mention that I used two AI models quite a bit, I only used them for parsing logs and deciding what to do. All of the code is actually handwritten.
https://redd.it/1xjx85h
@r_devops
