270
Reddit DevOps. #devops Thanks @reddit2telegram and @r_channels
For application configurations in OCI
you use Cloud Shell quite a lot? I'm new here and I'd like to know if people actually use it
https://redd.it/1tc843e
@r_devops
MCP servers just showed up in our infrastructure and I genuinely have no idea how to secure them, anyone been through this?
Not panicking but definitely out of my depth and i'd rather admit that now than figure it out after something breaks.
I've been doing DevOps for about three years at a mid-sized SaaS company. pipelines, containers, infra automation, the usual. last month our engineering team started integrating MCP servers to power some internal AI agent tooling and it landed in my lap to manage the deployment and infra side of it.
The problem is that everything i know about securing infrastructure doesn't map cleanly onto this. i can lock down a container. i can harden a CI/CD pipeline. but MCP is a different thing entirely. the servers expose tools that AI agents can call autonomously, and some of those tools have filesystem access, shell execution, database connectors. the blast radius of a misconfigured permission scope here feels genuinely significant and i don't have a framework for thinking about it systematically yet.
What's been keeping me up is the agentic side of it. these aren't just APIs sitting behind auth. the agents decide what tools to call and chain them together without a human approving each step. our current pipeline validation has already started flagging permission scope warnings on three of the deployed MCP tools and i blocked the deployment because i didn't know what the acceptable threshold even was. i've been piecing things together from blog posts and the handful of MCP security write-ups that exist but nothing gives me a repeatable methodology i can actually build a process around.
https://preview.redd.it/ay2kuyzyrw0h1.png?width=648&format=png&auto=webp&s=4b178a766229d4914c4927874e1c81e9757aa850
This is basically what my week has looked like. pass rate dropped from 96% to 81% since we started integrating MCP servers and almost all of the failures are permission or schema validation errors i don't fully understand yet.
Has anyone here gone through this? specifically curious whether there's any structured training that actually covers MCP security mechanics rather than AI security broadly, and how you're handling scope definition in your engagement agreements when the blast radius of these servers isn't obvious even to the people who built them.
https://redd.it/1tc01ui
@r_devops
Deployment advice for early stage startup!
Hello everyone,
We are running a small startup and the problem I am facing right now is single point of failure. Since we don't have much budget, we have hosted in cheap VPS as of now.
We have multiple services(python, node, db, redis, etc) and everything is dockerized inside a compose. So we run staging and production environment behind a nignx revere proxy. Both environment is hosted in single vps. We don't have any monitoring and observisibilty tool right now. The way we deploy is build docker image via github action and push it into vps and run it.
So for our setup, how can we improve our deployment and what are the best strategies we can adapt.
Thank you.
https://redd.it/1tc0h4j
@r_devops
We rebuilt infrastructure from backups as a DR-test. The restore worked. The environment didn’t.
Recently we rebuilt infrastructure from backups while setting up a new environment.
Part of the idea was also just seeing how recovery would actually go in a real disaster situation and what kind of hidden problems would show up along the way.
Luckily this wasn’t a production outage, so nobody was panicking and we could take our time digging through issues properly.
We thought it would take maybe a couple of days. It ended up taking weeks...
Every few hours we discovered something new: forgotten settings, incompatible software versions, undocumented dependencies, random unexplained errors, or some component nobody had touched in years.
The good part is that the next test restore was dramatically faster because we already understood most of the weak spots and had documentation for the recovery process.
https://redd.it/1tbuc4x
@r_devops
How are you picking which GitHub action to use from the marketplace?
My initial question was about an action to show the terraform plan as a PR comment to aid reviewing in r/terraform. This community does not allow x-posts, so I'm making a new one based on what I learned, but I think the lovely people who discussed are lovely and that you should read it
https://www.reddit.com/r/Terraform/comments/1t4j5pi/how\_do\_you\_pick\_github\_actions\_so\_many\_similar/
Having said that, question for you:
If you have a need, usually the first thing we think is "I can't be the only person with this need - let me see what exists"
(and if you're like me and don't find a pile of things - I think HOLY COW I AM REALLY LOST IF NOBODY ELSE NEEDS THIS)
I found that many of the actions in the marketplace sorted by popularity were abandonware.
I'm not a really strong programmer so I'm leery of the brand new, some seem like one-shot AI commits. Even if they're not 'AI slop', they aren't likely to be a maintained project?
Some actions are just a hundred lines of yaml, and others solving the same problem have unit tests , and end to end tests - is that necessary? or is it polshing canon balls?
I hope this is far enough from a x-post to be allowed. This is a bigger community and I want perspective
I used to only do 'published from a vendor I know', but there are some good things out there from small teams or even solo devs?
https://redd.it/1tbndcs
@r_devops
CI & Release environments automation
https://github.com/sparshev/aquarium-fish/wiki/Aquarium
https://redd.it/1tbi8o2
@r_devops
How are you managing EC2s for ECS clusters?
I am currently using 2 variations, both manual EC2 provisioning and also using a Capacity Provider with an ASG.
In my case, both work well, and can't really pinpoint many pros and cons of each (other than the auto scaling aspect of course, which is not that beneficial unless the hosted services support auto scaling themselves, IMO). Perhaps maintenance is a little bit easier when using an ASG, since you can easily scale down and up while changing AMI, etc...?
What are you thoughts and experiences about this?
https://redd.it/1tbfz12
@r_devops
Integration issues with TLS
What is the most weird thing you have faced with 3PPs integrating into your system? mine is mTLS. This particular provider couldn't even consult Google but said their company policy barred sharing screens it took 4hrs for exceptional approval to bypass the policy and 10mins to solve the issue
https://redd.it/1tbdfwc
@r_devops
A way to pin GitHub Actions by SHA without losing Dependabot vuln alerts; is this the best way?
I'm figuring out how to pin all my team's third-party actions by commit SHA (assuming no immutable releases available) after the recent supply chain attacks. But this comes with 2 problems:
1. Dependabot security alerts don't work on SHA-pinned actions.
2. We have lots of repos, so hash management across all of them becomes painful very quick.
Renovate and Dependabot version updates don't help much. They bump the SHA to whatever the latest tag points to, which gets you back to trusting semver tags.
Here's what I came up with:
Step 1 - Create a repo of thin wrapper actions around the third-party ones. Pin the external dependencies by SHA inside the wrappers. In your workflows, call the wrappers (pinned by branch or semver). Now hashes live in one place.
Step 2 - a "dummy" workflow purely for Dependabot. Create a workflow you never run, which contains all the external actions there pinned by semver tag corresponding to the SHA you're actually using. Dependabot security alerts start firing again.
You have to maintain consistency between the dummy workflow and the actual internal actions you used; I guess maybe Claude Code can help here. It feels janky though. Any better alternatives that I should think of?
https://redd.it/1tb6cd1
@r_devops
Looking for agentic/cheap post-deploy validation tools — what's actually out there?
Trying to figure out what's available before I roll my own.
Setup: event-driven microservices, deploys every other day. After each deploy I end up opening 10-15 dashboards to verify the event pipeline is still healthy — consumer lag, error rates per service, downstream effects.
Already tried alerts: set up anomaly-based alerts thinking it would replace the dashboard ritual. Ended up being either too noisy (firing on normal traffic fluctuations) or missing real issues entirely. The core problem with alerts is they're always-on — they don't know a deploy just happened, so they can't distinguish "this metric shifted because of the deploy" from "this is just normal variance." So we're back to manually checking dashboards after every deploy anyway.
What I actually want: something deploy-aware — wakes up when a deploy happens, compares post-deploy telemetry against pre-deploy baseline, and surfaces anything that shifted significantly. Not looking for magic, just "error rate on this route was 0.2% before deploy, now it's 3% — here's a trace." Something that does the baseline comparison automatically so I don't have to hold it in my head across 10-15 dashboards.
What I've looked at:
Datadog Deployment Gates — looks closest but company won't buy Datadog just for this
Argo Rollouts AnalysisTemplate — requires defining every metric upfront, which is the thing I'm trying to avoid
Honeycomb deploy markers — visualization only, not autonomous
Middleware.io — full platform, didn't go deep
Looking for OSS or cheap/personal-use options. Or is everyone just eyeballing dashboards after deploys and pretending that's fine?
https://redd.it/1tb3uof
@r_devops
Salary stagnation?
I have recruiters who reach out semi-frequently. But every offer is in the 140-160 range for SRE maybe 170 if it’s a senior SRE role.
For hedge funds it’s usually around 250-300k. But these are the same numbers I have been seeing for years at this point.
Not sure if it’s just a result of my own personal bubble or if others are seeing the same thing? It feels like salaries have not meaningfully increased since covid times.
Roles are usually based in mid-high COL in the USA.
https://redd.it/1tazcfy
@r_devops
how do you classify production AI agents in your infrastructure web app, batch job, or something new?
ive been doing infrastructure work for around 7 years, last 18 months specifically running AI agent workloads in production. 5 agents, \~50M tokens/month combined, mostly on AWS with bullmq for queues and postgres for state.
the thing thats genuinely confusing me: AI agents dont quite fit any of the boxes i was trained in.
theyre not stateless web apps agents carry conversation state, intermediate tool outputs, retry context. running them like ECS web services means losing conversational state on container restart.
theyre not batch jobs they need to respond to webhooks in seconds, not minutes. cant treat them like cron jobs.
theyre not microservices too much state passing between services turns the orchestration layer into a distributed transaction nightmare.
what ive ended up with feels weird. agents run as long lived workers polling a queue, state lives in postgres + redis, every external tool call gets idempotency keyed at the edge, observability is split across cloudwatch (infra metrics) and s3/athena (llm input/output traces because cloudwatch costs got out of hand).
things i still dont have good answers for:
where should i set timeouts on agent runs? web app timeouts (30s) kill complex tasks. batch job timeouts (hours) hide infinite loops. middle ground feels arbitrary
whats the right SLO model for a non deterministic system? cant promise 99.9% correctness when the underlying model is probabilistic
how are people doing blue/green or canary deploys when agent behavior depends on prompt versions interacting with model versions
where do you draw the line between agent decides vs ops team approves ive been adding human in loop for irreversible actions but the policy feels ad-hoc
curious how others are handling this in production. are you treating agents as a new infrastructure pattern or forcing them into one of the existing boxes? specifically interested in how teams further along have structured their on call response for agent failures, since the failure modes look nothing like web app failures.
https://redd.it/1tat745
@r_devops
DevOps / Kubernetes / OAuth2 Proxy + Keycloak
Hi everyone,
I’m building a **local zero-trust authentication setup on Kubernetes (Docker Desktop on Windows/WSL2)** and I’m stuck on an OAuth2-proxy + Keycloak login loop / CSRF issue.
I would really appreciate any help or pointers.
# Stack
* Kubernetes: Docker Desktop (WSL2)
* Ingress: NGINX Ingress Controller
* Identity Provider: Keycloak (OIDC)
* Proxy: oauth2-proxy (Helm chart v7.15.2)
* Backend: FastAPI
# Local domains
All mapped in local hosts file:
* `fastapi.test` → protected application
* `auth.test` → oauth2-proxy (OIDC gateway)
* `keycloak.test` → Keycloak
# Intended authentication flow
1. User visits:`https://fastapi.test`
2. Redirected to oauth2-proxy:`https://auth.test/oauth2/start`
3. Redirected to Keycloak login:`https://keycloak.test/.../auth`
4. Login succeeds and redirects back:`https://auth.test/oauth2/callback?code=...&state=...`
5. oauth2-proxy should validate session and redirect user back to:`https://fastapi.test`
# Problem
After successful Keycloak login:
* I get:`Login Failed: Unable to find a valid CSRF token`
* Then:
* Redirect loop between `/oauth2/start`, Keycloak, and `/oauth2/callback`
* Eventually ends with **403 at** `auth.test`
# Cookie behavior (observed)
* CSRF cookie is created on `/oauth2/start`
* Session cookie is also created
* But after callback:
* CSRF validation fails OR cookie is not accepted/recognized
* Sometimes cookie domain changes or is cleared
# oauth2-proxy config (Helm)
Key parts:
provider: oidc
oidc-issuer-url: https://keycloak.test/realms/zero-trust-local-platform
redirect-url: https://auth.test/oauth2/callback
cookie-domain: auth.test
cookie-secure: true
cookie-samesite: lax
cookie-path: "/"
cookie-name: _oauth2_proxy
reverse-proxy: true
whitelist-domain: fastapi.test,auth.test,keycloak.test
Upstream:
upstream: static://200
# Ingress setup
oauth2-proxy ingress:
* Host: `auth.test`
* Path: `/`
* NGINX ingress controller
* TLS enabled via cert-manager (self-signed local CA)
FastAPI ingress uses:
* [`nginx.ingress.kubernetes.io/auth-url`](http://nginx.ingress.kubernetes.io/auth-url) pointing to:`http://oauth2-proxy.oauth2-proxy.svc.cluster.local/oauth2/auth`
* `auth-signin`:`https://auth.test/oauth2/start?rd=$scheme://$host$request_uri`
# Keycloak config
* KC hostname: `keycloak.test`
* Proxy mode enabled via:KC\_PROXY\_HEADERS=xforwarded KC\_HOSTNAME=keycloak.test KC\_HTTP\_ENABLED=true
* Running behind ingress with TLS
# What I already tried
* Chrome + Edge + Incognito
* Changed cookie-domain (`.test`, `auth.test`)
* Changed SameSite (`Lax`, `None`)
* Verified redirect URI matches Keycloak client config
* Verified OIDC issuer endpoint
* Checked ingress annotations
* Verified reverse proxy headers
# What I suspect
I think this is related to one of:
* CSRF cookie not surviving redirect chain
* cookie domain / path mismatch between:
* `/oauth2/start`
* `/oauth2/callback`
* reverse proxy headers missing (`X-Forwarded-*`)
* oauth2-proxy state/CSRF binding issue
* browser SameSite + local domain behavior
# Question
Has anyone successfully run a **3-domain local OIDC setup like this**?
* fastapi.test (app)
* auth.test (oauth2-proxy gateway)
* keycloak.test (IdP)
If yes, what was the **critical config fix** that resolved CSRF / redirect loop issues?
https://redd.it/1tapyei
@r_devops
AI tools consistently misconfigure environment variables. Here's what to audit before deploying.
Environment variable handling in AI-generated Next.js projects has a predictable set of issues. Hardcoded values that should be env vars. Mix of `process.env.KEY` and direct string literals for the same service in different parts of the codebase. And most critically, `NEXT_PUBLIC_` prefixes on variables that should be server-side only.
The `NEXT_PUBLIC_` issue is the one worth flagging in every review. That prefix causes the variable to be inlined into the client bundle at build time. There's no runtime override. If a secret ends up there, rotating the key and redeploying is the only fix, and any version of the bundle with the old key is already in CDN caches and browser caches.
Standard checks I run: grep for `NEXT_PUBLIC_` and verify each one is intentionally public. Check that `.env` is in `.gitignore` with a committed `.env.example`. Confirm prod has different key values from dev.
What does your env var review process look like for new projects?
https://redd.it/1targ4i
@r_devops
Built an open-source AWS scanner that maps findings to SOC 2 controls and hashes every API response, looking for feedback from people who actually deal with this
Hey y'all, im a student. Built something AWS-specific over the past few months and wanted honest feedback from people who know this better than I do.
Quick background: I did this after speaking with 50+ pre-series A founders, auditors & CISOs. What I learned from these conversations: main problem with GRC tools is that the automation exists but systems are constantly changing so audits are slow or mistrusted. There isn't a way for auditors to verify with certainty that the evidence is correct. There's no chain of custody between a finding and the actual AWS state that produced it.
What it does technically:
Assumes a read-only IAM role via STS AssumeRole with ExternalId binding, fans out across 15 AWS services and 6 regions in parallel, maps findings to SOC 2 Trust Services Criteria controls, and stores every API response with the exact endpoint, timestamp, region, and SHA-256 hash of the raw response. The idea is that an auditor can re-run the same calls themselves and verify the hashes match, cryptographic chain of custody between the finding and the source. 30 seconds to deploy, 5 mins to run, completely for free.
A few decisions I'd genuinely like feedback on:
All calls are SigV4-signed from scratch, no AWS SDK. Runs as a Cloudflare Worker so the SDK wasn't an option, and doing it from scratch means every call is fully visible in the open-source code. Curious if anyone sees problems with this.
Concurrency is capped to avoid rate limits across parallel service calls. Using Cloudflare Queues for the AI analysis layer, per-control reasoning runs in separate queue messages with exponential backoff. Open to better approaches.
IAM role is scoped to SecurityAudit + ReadOnlyAccess with an explicit Deny on secret values. Provisioned via CloudFormation so engineers can read it before deploying.
What it doesn't do:
AWS control plane only, IAM, S3, CloudTrail, Config, EC2/VPC, KMS, GuardDuty, SecurityHub, RDS, Lambda, WAF, SSO. Not application data, not secret values, not anything outside configuration metadata. Doesn't cover full scope of Type l.
What I'm looking for:
Honest technical reactions, especially from anyone who's worked on IAM auditing, AWS security tooling, or has gone through a SOC 2 as the engineer who got handed the problem. And if anyone wants to run it against a sandbox account and tell me what's broken, that would genuinely help.
Repo: github.com/adog0822/AWS-Evidence-Layer
https://redd.it/1tafe9d
@r_devops
Need help understanding GoReleaser + GitHub Actions + Homebrew/Scoop automation for my CLI tool
I built a Go CLI tool https://github.com/aminshahid573/gen
Now I am trying to set up proper distribution (Homebrew, Scoop, releases), but I’m honestly confused about how the whole pipeline works.
Right now I used AI to generate my setup, but I don’t fully understand it and I feel like I’m just copy pasting things without knowing what I’m doing and not aure is that working or not..
What I want is..every time I create a new version tag (v1.0.0, v1.0.1, etc.) gitHub Actions should automatically:
1. run tests
2. build binaries for Linux/macOS/Windows
3. create a GitHub release
4. update Homebrew tap
5. optionally update Scoop manifest
---
my current setup is .github/workflows/release.yml
https://github.com/aminshahid573/gen/blob/main/.github%2Fworkflows%2Frelease.yml
---.goreleaser.yml
https://github.com/aminshahid573/gen/blob/main/.goreleaser.yml
---
Upto i created GitHub repo for CLI, reated Homebrew tap repo (homebrew-gen) and Wrote formula manually
I am confused that do I actually need HOMEBREW_GITHUB_TOKEN or is GITHUB_TOKEN enough?
how does GoReleaser exactly update Homebrew tap repo?
or do I need to manually create formulas or GoReleaser handles everything?
What is the correct minimal setup for github Actions,GoReleaser,homebrew tap, scoop bucket .
If someone can explain correct clean workflow from scratch and minimal working config, what each part actually does
that would really help me understand instead of just copy pasting AI configs.
https://redd.it/1tc506c
@r_devops
Open-source byte-exact dedup tool — EOSE Labs picked it up as a Terraform state hygiene layer for their sovereign language fleet
Disclosure: I'm the author. Posting because the use case isn't one I would've pitched to r/devops on my own, but a production team picked it up for exactly that and figured this sub might find it useful.
**The case study**
EOSE Labs ([pemos.ca/pemgraphs](https://pemos.ca/pemgraphs)) built a sovereign infrastructure architecture they call PEMGRAPHS — eight language nodes (Python, Rust, Go, TypeScript, Bash, Perl, Lean 4 for formal verification, HCL/Terraform), AKS + Azure Key Vault, multi-tier ADA vault secret rotation, 3,051-theorem Lean 4 proof engine. They're using my dedup tool (Merlin) at the Terraform state node and the Python v4 lineage tier as a hygiene primitive.
Their public shoutout: *"Corbenic didn't know they were unlocking sovereign fleet context optimization. They were just building really good dedup."* [pemos.ca/community-shoutout](https://pemos.ca/community-shoutout)
**What the tool is**
* Byte-exact line-level dedup using xxHash3-64. Deterministic, lossless.
* Single-threaded, \~250 KB Windows x64 binary, statically linked (no msys2 / runtime deps).
* 1.10 µs median latency per chunk (their measurement, not mine).
* MIT-licensed integrations: CLI, MCP server, HTTP proxy.
* Hard-enforced caps in community tier: 50 MB/run · 200 MB/day · 2 GB/month. Refuses oversized work cleanly.
**Why DevOps might care (beyond AI workloads)**
* Terraform state files balloon fast in large infra. EOSE uses it as a hygiene layer at the HCL/Terraform node.
* CI/CD log pipelines accumulate duplicate noise. `./merlin-lite input.log --output-dedup=clean.log` is a standalone primitive.
* Anything where you process append-heavy streams and want a fast pre-archival pass.
**Install**
curl -LO https://github.com/corbenicai/merlin-community/releases/latest/download/merlin-community.zip
unzip merlin-community.zip && cd merlin-community
python shared/install_helpers.py <integration> enable
Or just use the binary standalone positional input file + `--output-dedup=PATH`.
**Honest caveats**
* Line-level exact-match. Not semantic, not fuzzy. If you need near-duplicate detection on rephrased content, this isn't it.
* Open-core: there's a closed-source Pro engine for high-throughput servers. What's in the public repo is what runs in the community edition same MCP interface, same caps.
* Windows x64 binary in v0.2.1 release. Linux + macOS coming once cross-platform CI is up.
Repo: [**github.com/corbenicai/merlin-community**](http://github.com/corbenicai/merlin-community)
Zero telemetry. Curious if anyone here has Terraform state hygiene pain that this might fit, or if EOSE's pattern (using a dedup primitive as a building block in a bigger system) maps to something you're already doing.
https://redd.it/1tc1rbg
@r_devops
How do you deal with engineers who refuse to touch the actual workflow/process side?
I have a couple really strong engineers on the infra/platform side who are honestly great technically. Fast problem solvers, reliable during incidents, know the systems deeply, people trust them. But they absolutely hate anything that looks like process maintenance. No ticket updates, no documenting changes properly, no ownership notes, no updating runbooks after incidents, no cleanup of monitoring alerts, barely any visibility into what is changing unless you directly ask them.
Their mindset is basically the systems work, thats what matters. The problem is everything becomes tribal knowledge very fast. During incidents half the context lives inside specific people’s heads. If somebody is out, suddenly simple operational things become detective work because nobody knows why something was configured a certain way 8 months ago.
And I get their side too honestly. A lot of devops work already feels overloaded with tooling, alerts, dashboards, pipelines, permissions, reporting, tickets etc. I understand why engineers want to spend time fixing systems instead of updating 4 different platforms explaining that they fixed the systems. But at the same time the operational overhead for the rest of the team becomes huge when basic visibility is missing.
I tried lighter processes, simpler templates, reducing required updates to almost nothing, explaining the bus factor issue but eventually everybody slowly drifts back into just message me directly if you need context.
How other people handle this balance without turning good engineers into full time administrators?
https://redd.it/1tbvvxn
@r_devops
When have you used Terraform in a DR scenario?
I’ve been in the industry for almost 7 years now and worked at 4 separate companies. In that time, I have not been in a single situation where I’ve had to rebuild an environment or part of it using Terraform/OpenTofu/Pulumi.
This post is not against IaC, that would be egregious. It’s just that one of the many use cases of IaC is DR, but I’ve never experienced it or come across it. Have you?
https://redd.it/1tbp5wu
@r_devops
New to DevOps – What Should I Learn First & What Does Your Daily Work Look Like?
Hi everyone,
I’m exploring DevOps as a career path. I’m new to this field and trying to understand how to start properly.
I wanted to ask experienced DevOps engineers:
1. What should a beginner learn first in DevOps?
2. Which tools are most important for freshers (Linux, Docker, Kubernetes, AWS, Jenkins, Terraform, etc.)?
3. How much scripting is required (Bash, Python)?
4. What does your day-to-day work look like in a company?
5. Do DevOps engineers mostly work on deployments, monitoring, CI/CD, cloud management, or something else?
6. What projects would you recommend for building a strong resume in DevOps?
7. Any mistakes beginners should avoid?
I’d really appreciate practical advice from people already working in DevOps.
Thanks!
https://redd.it/1tblnnp
@r_devops
Why up-sizing nodes usually doesn't fix Kubernetes P99 spikes
Lately, I’ve been looking at large clusters where the default answer to P99 spikes is vertical scaling. Teams throw more cores at the problem to give apps room to breathe, but it often fails to solve the root cause.
We're testing a layer that allows the kernel to prioritize execution based on the specific runtime needs of each workload. Instead of treating a critical database and a background scanner the same, we give the kernel the context it needs to prioritize execution in real-time.
In our lab tests, P99 latency for Redis and Nginx dropped by about 85 percent and database throughput increased by roughly 60 percent. This happens beneath the app layer, so there are no sidecars or code changes.
I’m curious if this resonates with your experience.
Do you up-size nodes just to stabilize graphs even when utilization is low?
Would a read-only report showing exactly where your node is fighting your hardware be useful for your team?
We are looking for one or two real-world environments to validate our data. We have a non-intrusive Observe Mode that just monitors signals and generates a report without changing any scheduling. If the data shows clear potential for improvement, the logic can move into an active mode to fix those bottlenecks automatically in runtime.
Feel free to ping me if you want to chat or see the technical benchmarks. I’m keeping this anonymous for now due to current contracts, but would love to hear more about real use cases and pains!
https://redd.it/1tbi8s8
@r_devops
Moving away from building infrastructure for the AWS brand
My background is \~2 years of cloud engineering experience in consulting. I've mainly administered k8s clusters and built Terraform modules and designed Aws architectures.
I now have an offer as an AWS Cloud Support Engineer for a newly launched region. Because the region is new, many issues are still internal. However I won't be actively building Infra myself. Instead I'll be debugging customer infrastructure issues and escalating internally. My motivation for the role is the AWS internals exposure, partly the brand on the CV for long term career leverage. I want to return to hands-on cloud engineering after this chapter. I would obviously continue with Open Source next to this job and build some demo setups with Terraform. However I am wondering whether trading 18 months of active building for AWS brand and internals knowledge a reasonable calculated bet? Or will it be tricky to move back into a hands on engineering role?
https://redd.it/1tbcz5b
@r_devops
DevOps engineer with an AI-ish stack — what should my next steps be?
Hey r/devops,
I'm a DevOps engineer with a few years of experience trying to level up my AI integration skills and would love some advice from people who've been down this road.
Here's my current stack:
\- alibaba Cloud - (not by choice, RIP me)
\- argoCD for GitOps
\- gitlab + gitlab pipelines for source control and pipelines
\- Open WebUI with Claude and ChatGPT agents running through LiteLLM
so you can say I'm not starting from zero, I've got ai models accessible to my team and an MCP server up. but I feel like I'm scratching the surface of what's possible and I'm not sure what to tackle next.
some directions I've been thinking about:
1. AI-assisted gitlab pipelines (MR reviews, release notes, security scanning via LiteLLM)
2. Connecting MCP tools to our actual infra (query Alibaba resources, trigger pipelines, check argoCD sync status)
3. Going deeper on MLOps — learning to deploy and monitor models properly
What would YOU prioritize in my position? Is there something that could add value to me I'm missing? any courses, projects, or resources that actually helped you (not just theoretical stuff)?
Appreciate any advice 🙏
https://redd.it/1tbbobf
@r_devops
Need Suggestion for Centralise logging system
Our company is running 20+ EC2 servers on an autoscaling group. Currently, developers are login into each EC2 instance to see logs. autoscaling creates a delete EC2, so how will I manage the logging system, how will they be able to send logs to centralise logging application?
What logging tool are you using at your company?
We are looking for an affordable option that we can host on our own server. We don't want to buy a subscription if we can get a good free solution.
https://redd.it/1tb6jyh
@r_devops
How do I get into devops
I’m current an on prem sysadmin focusing on endpoint, with entra and intune with a tiny bit of azure thrown in, my sr does the routing firewall and switching stuff
I do a lot of app packing, scripting, reports, graphAPI and taking apart application installers so we make them run in our endpoint environment, I also deploy scripts and have a deployment pipeline I made
Trying to work out where to go on my career program, I don’t have a degree! Self taught!
https://redd.it/1tb1h2m
@r_devops
How do EU companies think about dependency on US hyperscalers?
EU-based companies: how are you thinking about dependency on US hyperscalers?
Curious how this is handled in practice across different industries.
For those working in EU-based companies:
* Are most/all of your workloads running on AWS/Azure/GCP?
* Do you have a realistic “Plan B” if you ever had to reduce dependency on a hyperscaler?
* If yes:
* How is it structured?
* Multi-cloud?
* On-prem fallback?
* EU providers?
* Kubernetes/portable infra?
* Data replication only?
* If no:
* Is it because there’s no regulatory/business pressure?
* Or because the operational/financial tradeoffs simply don’t justify it?
Not looking for political takes or marketing answers (Basecamp DHH story) - more interested in real-world operational thinking from DevOps/Platform/Infrastructure teams.
Would especially love to hear perspectives from:
* fintech/banking
* healthcare
* public sector
* SaaS companies
* startups vs enterprises
https://redd.it/1taxpi1
@r_devops
Devops internship resume advice
https://redd.it/1tavux1
@r_devops
Good Chainguard alternatives for base images
We’ve been evaluating Chainguard images for reducing CVEs in our base containers, and overall the approach makes sense. Things like smaller images, fewer packages, less to patch. That said, pricing and flexibility are starting to come up as concerns, especially as we scale across more services.
I’m curious what others are using as alternatives. Are you sticking with minimal images like Alpine or distroless, building your own, or using something else entirely? Main goal is to reduce CVE noise without creating more overhead for engineering.
https://redd.it/1tatilm
@r_devops
GitLab's "Act 2"
https://about.gitlab.com/blog/gitlab-act-2/
https://redd.it/1tai4sl
@r_devops
IAM drift keeps recurring… when do you turn a fix into a CI gate vs leave it as a runbook note?
CI deploy fails due to IAM drift. the Sr Engineer finds it, fixes it, closes ticket.
Weeks later, different service, different engineer, same root cause shows up again.
Not asking about tooling or documentation. here… assume monitoring + CI/CD gates already exist and log a lot of this “class” of problems.
What I’m curious about is :
When you moved prevention earlier (CI / deploy / monitoring gates), how did you decide what stays as “incident-time knowledge” vs what gets promoted into a hard pre-deploy check?
For example: if an IAM drift issue is discovered during a deploy, do you treat the fix as:
something you only add to the postmortem/runbook (so next engineer still has to recognize it), or As something that becomes a CI gate like “fail deploy if IAM policy diff ≠ baseline”?
https://redd.it/1tacoag
@r_devops