270
Reddit DevOps. #devops Thanks @reddit2telegram and @r_channels
Stuck Between IT and RKE2: Need Advice on Operating Without Sudo Access
>Hey guys,
>
>I am currently implementing an RKE2 setup in my DevOps team to handle deployments and deliveries for our application.
>
>The problem is that I won't have direct root access to the servers, so access to root-owned directories is also restricted. If I want to install anything, requests have to be raised to the IT team, and they will get it installed. (The thing I hate about MNCs they're too paranoid and too slow as well, iykyk.)
>
>But here's the frustrating part: the "installation" done by the IT team is often just placing the downloaded packages in my accessible directory... duh!!!
>
>So then I need to contact them again and have them actually complete the installation and setup using their root access. And even after the installation is completed, there will still be situations where I'll need elevated privileges to manage pods, services, deployments, and other operational tasks.
>
>I can't exactly request sudo access, so at this point I honestly don't know what I'm supposed to do or what request i need to raise as well.
>
>My requirement is very minimal for now, as I am only experimenting with this(but will be mandated in future). I won't be connecting remote agents, just a single server, and based on my application requirements I narrowed the packages required to the following:
>
>
>
>rke2-images-core.linux-amd64.tar.gz
rke2-images-cilium.linux-amd64.tar.gz
rke2.linux-amd64.tar.gz
>
>
>
>Based on the documentation, I tried using customized paths and accessing the binaries directly to complete the setup, but I still got stuck at some point where root access was required. I don't want to go into everything I did to get there, so yeah, if you guys can suggest the minimum set of steps or activities that I can request from the IT team, which would allow me to effectively use and manage this RKE2 setup afterward, that would be really helpful.
>
https://redd.it/1xj9ajw
@r_devops
Binance hack and a DevOps laptop
https://redd.it/1xj4flc
@r_devops
Passed the AWS Cloud Practitioner (CLF-C02) exam — sharing my study plan, resources & pro exam tips!
https://youtu.be/fH7CaAw--vc
https://redd.it/1xiivfg
@r_devops
DevOps conferences in Europe 2027
Hi everyone,
What are some quality conferences in Europe worth visiting in 2027?
https://redd.it/1xijx97
@r_devops
I landed a "devops" role as a junior
30K\~ DAU, I have full ownership over an infra under AWS/multiple on premise servers.
we use ECS, terraform, instead of ansible I use a deploy.sh that i give to my boxes with terraform, no graceful deploys. we have our own orchestrator (K8S like) for stateful containers that we have to spin up and down very often (median 1 min of uptime per container) with unsafe code coming from AI/users.
I took the place of someone who left that company and explained me nothing, we are 2 devs, nobody understands how this entire infra works expect me. have been adding observability a lot using cloudwatch, multiple alerts so I page myself at the right times, it's stable but we need to scale to at least 5x (an objective the hierarchy gave me). it currently costs 3.5k/month in aws/on premise servers. btw it's not a classical rest api where you run stateless apis and that's it.
I just discovered how redis worked under a distributed system like this, it's quite good :) I have been a postgresql guy but redis has been a saver for our threat model.
I often read here that you should put everything : EKS + K8S, ansible, terraform, make HA, grafana etc.. but since I manage this entire infra I have been resilient to all of this. If I refactor everything right now they have nobody to do the backend (which is also what I do here) -> "startup" role where one guy has multiple roles.
for the terraform it's all AI, I have been doing a lot of modifications and have created a terraform plan under ci cd so I can review myself, I removed a lot of "destruction" like the database, i versioned the s3, versioned the terraform state, put it to a private s3 etc...
there have been big big security issues (xss, csrf, cookie stealing, untrusted code that owns the entire box, shared passwords, untrusted s3 location so you could overwrite other users...) that were there before I got here
nothing rly interesting maybe for you guys in this experience but I have learned a lot, it has been a crescendo experience where i'm getting to understand more and more which tools to put there because I am the one who owns it 100%, and have to also do features (backend) so not much time either.
https://redd.it/1xigylx
@r_devops
How do you contain your agents' accesses?
Hey !
I guess most of us have a lot of admin accesses to tools like AWS, k8s clusters, Datadog, GitHub etc.
How do you contain your agents to read-only or curated permissions? Without restraining your own accesses.
I do have some sort of hook as a guard + a setup on pi.dev that copy my config and extract the read-only accesses only to do so, but it's never completely opaque.
https://redd.it/1xid62t
@r_devops
Suggest a good laptop for DevOps under ₹90,000 in India
Hi everyone!
I'm looking to buy a new laptop with a budget of ₹90,000 for learning and working with DevOps.
My primary use cases will be:
Docker and Docker Compose
Kubernetes and Minikube/kind
AWS (EKS, EC2, CI/CD, etc.)
Jenkins, Terraform, Ansible and Linux
Running multiple containers and virtual machines
Programming and general development
I'm not looking for a gaming laptop specifically. My priority is performance, multitasking, Linux compatibility, cooling and long-term reliability.
I'd appreciate recommendations based on real-world experience, especially from people working in DevOps or cloud engineering in India.
Thanks!
https://redd.it/1xhn43x
@r_devops
building is so easy and verification becomes bottleneck now !!
now we all know .. building/coding is easy with lot of ai tools/coding agents.. and pressure we are seeing during the deployment cycles verification and human in the loop cycle..
and wondering how are you guys handling it? we have team of 15+ and multiple teams and using Asana agents they have long runing agents and produce features/bug fixes super fast.. but determinstic verification lacks..
https://redd.it/1xhs0o2
@r_devops
CloudFlare Tunnel with pre-auth (and multiple IdPs)
I am working on setting up CloudFlare tunneling for a SaaS product in order to get WAF in front of the application and potentially use the CDN for faster distribution.
In the process there was also a requirement to move over to using pre-authentication in front of our SaaS solution, so that all requests are authenticated before they even hit our servers (similar to Azures application proxy).
It was suggested to use CloudFlare access, but this seems to more targetted towards internal apps and replacing / addition to a VPN in order to get ZTNA.
What would be a good way be to setup this pre-authentication? To me it seems there is no CloudFlare solution that handles this, so either I have to setup something to handle this check on CloudFlare or find a different vendor.
https://redd.it/1xhnx2t
@r_devops
blacksmith vs depot vs namespace alternatives, anything else worth a trial?
we were on buildjet until it shut down in march, been back on stock github runners since and the builds are noticeably slower. team of about 25, linux x64 mostly with a bit of arm
blacksmith, depot, namespace and avrea all look the same on the pricing pages so i cant pick, and every comparison i find is written by one of them lol. also seeing warpbuild and runs-on mentioned
if you picked one and stuck with it, what made you choose it? or did you try a couple and bounce off one
https://redd.it/1xhq1sj
@r_devops
I recreated a Node2Shell RCE attack in the form of an interactive exercise
On 21 October 2026, Google released patched versions of its Brotli compression library for CVE-2026-111497, a vulnerability nicknamed Node2Shell. It is an out-of-bounds write in the Brotli decoder. Node and many Python stacks bundle their own copy of Brotli, so patching libbrotli on the host doesn't fix them.
I made an interactive, hands-on lab exercise on this vulnerability. The goal is to give engineers a general overview of the attack vector and the response. You'll exploit a service as an attacker, then switch roles to the on-call engineer: upgrade the runtimes, rebuild and redeploy the images, verify the fix, and rotate the deploy token the attacker read from memory.
I did my best to highlight all the details, but if you notice any issues or inconsistencies, please let me know!
https://ransomleak.com/exercises/http2shell-rce/
https://redd.it/1xhohrx
@r_devops
Preparing for OCTA and CCA
Hey guys,
I'm preparing to get the CCA and OCTA cert to become golden kubestronuat, i see that the kode cloud cuurses are 20 hrs each, is it really necessary to go with that courses? also i saw that kyverno kode cloud course is so deprecated, is this true? i need to get them fast because cks will expire and i dont want to do it again xD
Thank you so much.
https://redd.it/1xhlud8
@r_devops
Don't hate the player, hate the game
https://preview.redd.it/pc09vwrur5yh1.png?width=1024&format=png&auto=webp&s=d2c85cafa5afef5aa28693dd9b74bc643a8029fb
my week in a meme
https://redd.it/1xhhbwh
@r_devops
How do small teams actually debug production issues without an SRE? (asking because I’m building something for this)
Genuine question for anyone on a small team (2–10 engineers) where the people shipping features are also the people getting paged.
What I kept running into: an alert fires, and the alert is the easy part. Then comes 45 minutes of jumping between logs, traces, the error tracker, and the code, trying to work out whether the checkout failure is the problem or just a symptom of a database timeout somewhere else. Then you push a fix, and honestly, half the time you’re not sure it worked. You just notice the errors stopped and hope.
That last part bugs me the most. “I deployed a fix” and “the issue is resolved” aren’t the same thing, but most workflows treat them that way.
Full disclosure: I’ve been building a tool around this called Relia. The idea is that it pulls the relevant evidence together, gives you a probable root cause (and says when it isn’t confident), suggests a fix plan, and then checks production after deploy to see whether the original error actually went away. It can also pass that context to AI coding assistants through MCP, so you’re not copy-pasting stack traces into chat.
I’m not linking it here because I’m more interested in whether my assumptions are right:
1. How long does it typically take you to go from “alert fired” to “I know the actual cause”?
2. Do you explicitly verify fixes after deploying, or just watch for it to recur?
3. If you’ve tried AI debugging tools, where did they fall short? My guess is they confidently make things up when they lack context, but I’d like to hear real experiences.
Happy to share what I’ve learned building it if anyone’s curious. Mostly I want to hear how other people handle this, because I suspect a lot of us are just winging it.
https://redd.it/1xhfl28
@r_devops
why is everyone still using cloud llms just to route tool calls? tested local cpu embeddings vs bm25 on 100k tools
honestly i dont get why current multi-agent setups are built like this.
if you connect 4 or 5 mcp servers (github, postgres, filesystem, slack etc) you easily end up with 40-50+ tool schemas. right now the standard pattern in langchain or claude code is dumping all 4,000+ tokens of json schemas into the prompt on every single turn just so the llm can pick which function to run.
you end up paying token costs and waiting \~2 seconds of network latency just for what is basically an L7 routing lookup.
i spent the last few weeks testing if we can just decouple tool discovery completely from the llm using a local cpu vector mesh (all-MiniLM-L6-v2 + chromadb hnsw index) and a split execution daemon.
instead of embedding just the function name (which fails on synonyms), i combined the tool name, description, schema keys, and tags into a single profile vector. when a prompt comes in, it embeds the query on cpu and pulls the top match before ever touching the llm.
ran a cold-cache benchmark across 100,000 synthetic tool endpoints on my local machine (standard cpu, no gpu):
method top-1 acc avg latency p95 latency
\---------------------------------------------------------------
naive keyword scan 38.1% 46.41 ms 57.60 ms
bm25 lexical index 40.4% 194.05 ms 247.25 ms
local vector mesh 70.7% 9.56 ms 11.44 ms
in-memory fast path n/a 0.082 ms 0.15 ms
https://redd.it/1xgsv6a
@r_devops
Github Account Got Suspended
It has been almost 3 days.. I was actively pushing my repos and working hard to create a opensource project. During this crucial time my account got suspended and now I'm really don't know what to do next.. do you guys faced such issues or do anyone know how long will it takes to unflag your account?
https://redd.it/1xj54ol
@r_devops
looking for a study partner to learn about devops
hello, I am a beginner programmer that's looking to learn about devops. i've made a web app, and also am currently learning algorithms and python. looking to see if anyone knows what devops is essentially thank you
https://redd.it/1xiwurc
@r_devops
Any one from Delhi/Dwarka for INFRA coffee discussion this weekend.
Hey, I'm new in the city, and looking for likeminded folks to chat and discuss current INFRA change happing.
https://redd.it/1xijzfm
@r_devops
Dealing with Colleague's AI-generated Code Reviews
As the title says, a new colleague recently joined our team, and he uses AI for everything. Not a single Slack message of his comes from his fingertips unfiltered.
The real pain point has been PR reviews. Our team is small, so reviews are hard to come by. I expected that having another teammate would make things smoother since we could share the review load, but his reviews don't add value, and they actually slow down our development cycle more than before.
I like to break my changes into small, digestible chunks that are easy to review. The problem is that he runs Opus 5.5 on xhigh against every change, and it nitpicks things that aren't even in the PR. It could be a two-line documentation change, and he'll comment that some existing code elsewhere in the same file, outside the scope of the change, needs to be modified. He then marks the pull request as "Changes requested" and doesn't re-review it for days on end.
How would you go about discussing this kindly with him?
https://redd.it/1xij5hc
@r_devops
Late 30's IT dude , looking for honest review
https://redd.it/1xi28j1
@r_devops
Kestrix: CI/CD pipelines defined in HCL outside the repositories, with DAG-based triggers
https://redd.it/1xid1m5
@r_devops
Devops with AI..
I've recently met a former colleague who currently works as a DevOps engineer at a company that forces its employees to use AI (Claude, I guess). Each employee has an AI budget that he HAS TO use every month. I believe the goal is increasing productivity per employee.
I have a lot of concerns about this approach to work, especially from the perspective of security, privacy, data leaks, and having a non-open-source AI app installed locally. What if the tool is hacked? What if my credentials are leaked? What if AI hallucinates and deletes non-recoverable data?
On the other hand, I understand this might be the new industry standard. If yes, I don't think there's a point of resisting change since the train will leave without me.
I would like to see what the perspective of other industry experts is and how they navigate this topic. Are you an early adopter? Or do you prefer to wait until the storm passes?
https://redd.it/1xhl0h8
@r_devops
The $87K API key leak had nothing to do with .env
Every time someone posts about a leaked API key, the top advice is the same. Put it in .env, add .env to .gitignore, done.
After the OpenCode breach and the LiteLLM mess this month I went back through the leaked key stories people posted this year. Found 10, with bills from $2 to about $87K. Not one was "I pushed my .env to GitHub".
Where the keys actually were:
\> In the JS bundle. A VITE_ variable in a gitignored .env, repo scanner said clean, then the build shipped it to every visitor. One scan of 978 AI-built apps found 37 doing this.
\> In an agent config file. LiteLLM's master key sat in opencode.json, got force pushed over but never revoked, and was used 136 days later.
\> In someone else's tool. Keys saved in OpenCode's console got drained, one person lost $5,200 of company credits.
\> In a WordPress backup plugin. One root AWS key reused across sites, tested on Aug 23, used 56 days later for about $8K of Bedrock.
\> In old Google keys that quietly gained generative AI access. That's the $87K one.
\> In nobody's leak at all. Someone got into an Anthropic console through email login and created 3 new keys, $5,473 in a few hours.
And then last week Railway got hit. Attackers decrypted customer environment variables straight from the platform, and Railway told everyone to rotate every key. Gitignored, never pushed, stolen anyway.
What actually helped was boring:
\> Hard spending caps. The one person who had one lost a few minutes of usage and that was it.
\> An alert isn't a cap. The $87K team's budget alert fired at $550, five hours in.
\> Revoke, don't clean up. Force pushing doesn't un-leak anything.
\> Delete keys you don't use, and know where every key lives (Infisical, Vault, whatever you already have).
\> Don't count on a refund. The $87K bill got $55K off, they still owe about $32K.
I'm not saying stop using .env locally. I'm saying "it's gitignored" isn't the end of the conversation.
https://redd.it/1xhoqdk
@r_devops
How do you prevent analytical queries from tanking your transactional database?
I was checking out how to build a live operational dashboard for a logistics app and found out about composed vs. converged HTAP architectures.
I’ve always been warned never to run heavy analytical aggregations on the primary transactional DB because the contention spikes your transaction latency. But a guide I was reading explained how HTAP systems are handling this now. A “converged” system runs both through the same engine (which sounds risky for resource contention), while a “composed” system actually separates the compute engines and keeps them in sync through shared storage or replication, so each side stays optimised for its own workload.
Apparently, you’re supposed to measure p99 transaction latency while the analytical queries are running, not separately.
If you are running live analytics on operational data, how are you keeping the two from interfering with each other?
https://redd.it/1xhtfxb
@r_devops
Interview tomorrow (5+ YOE DevOps), CV deep-dive, but I’ve been caught in reorg purgatory doing nothing for months. How do I spin this?
I have an interview tomorrow for a DevOps Engineer position (I have 5+ YOE). The recruiter explicitly told me the interview will be a deep-dive through my CV.
I’m in a massive bind regarding my current/most recent role and need some urgent advice on how to frame it.
**The Situation:**
* **Early this year:** I was the lead engineering resource tasked with getting a major product production-ready. The literal *day* it was ready, a massive reorg hit. The product was stripped from me, and most of its target end-users were laid off.
* **Immediately following:** I was pivoted to head up another large project for a different internal product. I busted my ass, completed it successfully, and my manager was impressed. Then... *another* reorg hit. This product was also taken away.
* **The past 5+ months:** My team was merged with another to form a new SRE team for a completely separate product. The scope was entirely undefined. In fact, one colleague's sole task for the last 4 months was just *figuring out what our responsibilities are*.
Because of this reorg limbo, **I have had zero actual engineering work to do for months.** I’ve repeatedly met with my manager and skip-level to voice concerns, but I just get generic corporate hand-waving about a "clear roadmap ahead."
**The Dilemma for Tomorrow:**
If they deep-dive into what I am doing *right now*, I see three options, and they all feel terrible:
1. **The Brutal Truth:** Admit I’ve been sitting on my hands doing nothing due to corporate dysfunction. (Worried this looks toxic or makes me look unmotivated).
2. **The Total Lie:** Pretend I’m still actively working on those two successful deployment projects and that they were never taken away. (Worried because my poor recall will make it obvious I haven't touched the code recently, and I'll have to fabricate metrics, current challenges, and future roadmaps).
3. **The Middle Ground:** Pretend I *just* transitioned off those projects a few weeks ago (similar issue as option 2).
How do I handle this in a technical deep-dive without sounding like a liar or a slacker? Has anyone successfully spun "reorg purgatory" to a hiring manager?
Thanks in advance!
**TL;DR:** 5+ YOE DevOps engineer with a CV deep-dive interview tomorrow. This year, I built and completed two major projects, but back-to-back corporate reorgs stripped them away. I've spent the last 4 months in team-merger purgatory doing absolutely nothing. Do I tell the truth, lie and pretend I'm still on the projects, or find a middle ground without looking like a slacker?
https://redd.it/1xhpyph
@r_devops
Seeking Guidance from DevOps Professionals | Career Transition Advice
I have 5 years of overall IT experience, including 3.5 years in Application & Production Support and experience in Storage Administration.
My current skill set includes application monitoring, incident management, troubleshooting, SQL Server, Windows Server, IIS, AppDynamics, and Linux fundamentals. I am also an AWS Certified Cloud Practitioner.
I am interested in transitioning into DevOps or Cloud Engineering and would love to learn from professionals who have made a similar career move.
I would appreciate your guidance on a few questions:
• Is it realistic to transition from Application/Production Support to DevOps with my current experience?
• Which skills should I prioritize to become job-ready: Linux, scripting, Git, Jenkins, Docker, Kubernetes, Terraform, or AWS?
• Would you recommend moving directly into a junior DevOps role, or transitioning through a Cloud Support, Platform Support, or SRE role?
• How much hands-on project experience is typically expected by recruiters when considering candidates transitioning from support roles?
I am ready to put in the work and build practical skills alongside my current job. I would appreciate honest advice, learning resources, or insights from anyone who has made a similar transition.
https://redd.it/1xhnl2s
@r_devops
Best Masters Program
Got my undergrad in EE, doing CI/CD and DevOps work now that I love and want to grow in. ideal graduate programs would be:
1. Takes job experience as waivers for prerequisites I may not have (EE math transfers but I lack CS classic education)
2. 8 week courses (give or take)
3. If CS isn’t the best MS to pursue I’d love to hear alternatives
Thanks to all in advance!
https://redd.it/1xhl6i4
@r_devops
At what point does monitoring become so noisy that engineers stop trusting it?
You add alerts because you want to catch problems early.
CPU spikes. Memory usage. Failed requests. Slow queries. Queue delays.
Eventually, the team starts receiving notifications throughout the day.
Some are important. Most aren't.
And that's when a dangerous habit develops: people start assuming the next alert probably doesn't matter either.
I'm curious how teams deal with this.
Have you ever deliberately removed a large number of alerts to make monitoring more useful?
How do you decide which alerts deserve to interrupt an engineer and which should just be visible on a dashboard?
https://redd.it/1xhevkl
@r_devops
Any beginner-friendly DevOps Discord servers or Slack communities?
Hey Everybody!
Does anyone know of any active Discord servers, Slack channels or study groups where beginners can connect, share resources, ask questions and support each other along the way?
https://redd.it/1xgw5xd
@r_devops
I got tired of having three cloud consoles plus an SSH session open during every incident, so I built one live topology for all of them. Lessons from six months of doing it.
Disclosure: I build Control Cloud, the tool mentioned halfway down. Everything else here is what I learned building it.
Context: I run infrastructure that is split across two public clouds and an on-prem rack, the way most mid-size shops end up after a couple of vendor decisions nobody wants to revisit.
Every cross-cloud incident was the same dance: provider A's console in tab 1, provider B's portal in tab 2 (after the conditional-access prompt), an SSH session into the on-prem box, and the monitoring tool to guess where the traffic went. The consoles don't see each other. A subnet that exists on one side but is routed to nothing on the other is invisible until production traffic hits it.
So over the last six months I built Control Cloud: a single live topology that polls every provider over its API and renders resources, networks and dependencies as one graph, with live status and cross-provider edges, and start/stop/reboot from the same surface.
Lessons that cost me the most:
1. Polling cadence has to follow how the cloud answers, not a round number. A full inventory crawl on one provider takes far longer than a UI should wait. We split it: the backend polls each region in its own goroutine behind a semaphore (unbounded concurrency = rate limits and context deadline exceeded), every region is published the moment it lands instead of waiting for all of them, and a file cache serves the last known state at startup so a restart isn't a blank screen. The UI refreshes every 10 s.
2. Dragged nodes must stay where the user put them. Auto-layout from scratch on every poll looks great in demos and is unusable in practice: people move nodes to match their mental model. A refresh updates status, not positions.
3. Provider parity is the whole point. If one provider shows nested scopes with subnets, every other provider has to show the same shape, even when its own model is flat. "The same concept renders differently per provider" is exactly what people are trying to escape. We keep a rules file derived from the code (the shared state helpers, the per-provider collectors, the navigator) and diff every change against it.
4. Status colors have to be honest. Every provider has its own vocabulary for "healthy" (RUNNING, AVAILABLE, Succeeded, Ready, Attached…). Inlining a regex per call site is how we shipped a red edge on a healthy cluster. One shared helper, one vocabulary per provider, and anything unconfirmed renders as unknown, never as a made-up warning.
5. Validation against live state paid back more than the canvas. Once everything is in one graph you can check a planned change against what is running: credentials with reach over the right scope, network (missing network or subnet = error; no NAT, no internet egress, no firewall = warning), on-prem hosts reachable, the dependencies of the template you picked. Anything it can't confirm comes back as informational, not as a green tick.
Stack: Go backend polling each provider SDK in parallel, React + Vite frontend, file-based cache, Postgres (Supabase) for orgs, audit log and schedules. Read-only API access with credentials you grant, encrypted with a managed KMS; no agents in your VMs. The Terraform it generates is committed to your repo and runs under your pipeline.
Live at https://controlcloud.dev/en if you want to poke around (free plan: 1 operator + 1 provider). Docs at /en/docs.
Happy to go deeper on:
- Why not the observability platform you already pay for (great at its thing; this answers a different question)
- The parity work: every provider added meant refactoring the first one
- Where the polling still doesn't scale (many regions per provider gets hairy)
Anyone here running their own version of this? What did you try first?
https://redd.it/1xgtio4
@r_devops
