270
Reddit DevOps. #devops Thanks @reddit2telegram and @r_channels
LiteLLM Compromised again - 1.107.0. PyPI Installs a Credential Stealer and Backdoor
The full article: https://safedep.io/litellm-pypi-compromise-credential-stealer-backdoor/
https://redd.it/1xbq9of
@r_devops
How do you simulate realistic user traffic in load testing?
We are looking to overhaul our load testing, since our current script only slams 10k users in 60 seconds and gives us garbage data. It exhausts connection pools and creates artificial bottlenecks that would not show under any kind of normal use case.
Real users ramp up slowly, and idle on pages, and don't jump from one to another without a second thought, they do not fire off database transactions with zero pause.
We should be reflecting real behavior, not inducing cold start failures.
Has anyone modeled this at scale? Should we make the step curves with warm up times to let caches and other systems warm up before hitting the expected load? We want to reflect real user behaviour, not just trigger cold start failures.
https://redd.it/1xblst1
@r_devops
I was thinking on competition about chess AI
What if we hosted a public tournament for chess bots to compete against each other and see which one comes out on top?
https://redd.it/1xb068c
@r_devops
Interviewing for a Software Support Engineer role (Linux, Python, Bash, CI/CD) — what technical questions should I prepare for?
Hey everyone,
I have a 30-minute interview coming up for a Software Support Engineer position at a defense contractor, and I'm trying to prepare as thoroughly as possible.
The recruiter mentioned there won't be a formal technical assessment, but I still want to be ready for any technical questions or troubleshooting scenarios.
For background, I currently work in cloud/data center infrastructure operations, where I use Linux, Python, and Bash for diagnostics, troubleshooting, and automation. I also have experience developing software applications, working with REST APIs, automated testing, and CI/CD.
Here's the relevant job description:
Responsibilities:
Support lab tooling and version control through CI/CD pipelines.
Create and deploy test scripts.
Support and troubleshoot software deployments on custom servers.
Support configuration management processes.
Develop unit tests and tailor rule sets.
Required qualifications:
Bachelor's degree in CS, Computer Engineering, or a related technical field.
2+ years of relevant experience.
Experience with Python and Bash.
Ability to independently and collaboratively solve complex technical problems.
Preferred qualifications:
Knowledge of CI/CD pipelines.
Experience with GitLab.
Experience with Linux and NFTables.
I'm particularly interested in moving toward DevOps/platform engineering, so this seems like a great opportunity.
For anyone working in DevOps, Linux administration, software integration, or similar roles:
1. What technical questions would you expect in an interview for this position?
2. What Linux troubleshooting scenarios should I be prepared to walk through?
3. How deeply should I study GitLab CI/CD, unit testing, and NFTables?
4. Does this sound more like DevOps, software integration, or traditional software support?
5. If you were interviewing a candidate for this role, what would you want them to demonstrate?
I'm not looking for shortcuts or leaked interview questions. I genuinely want to understand the technical expectations and make sure I can speak confidently about the work.
Appreciate any advice!
https://redd.it/1xaxlif
@r_devops
Kubernetes probes keep restarting services that just need more time
Hey folks, I've noticed readiness and liveness probes often end up with pretty similar settings, even tho they do different jobs. With tight timeouts and failure thresholds, a service that's slow to start or recovering from a dependency issue can get pulled out of rotation or repeatedly restarted.
I've seen probe settings get copied between services too, even when their startup times are completely different.
How do you guys set probe thresholds for services with unpredictable startup times or dependencies that occasionally respond slowly?
https://redd.it/1xatqcs
@r_devops
main skill u need for ur server stack?
The main skills knowing what services u need to run, not just picking Nginx or Apache, like I’m on BeAdmin rn, ran Webmin before. What changed ur setups u guys setup?
https://redd.it/1xah47j
@r_devops
J.Crew’s homepage is currently just a JSON health check for their "generative AI backend"
https://redd.it/1xa2353
@r_devops
What are your thoughts on people using AI to converse?
So we have this infra lead who pretty much uses ChatGPT to write up their replies to Slack messages. You can clearly see the use of AI in how their replies are structured. Because of this, sometimes, due to their AI’s lack of context, certain replies do not have the depth required for their level.
When I say use of AI, here’s some examples:
- The current 1TB threshold sits right at our normal free space, so the daily 50GB temp file dip trip it. Adding 200GB would just delay the same problem as data grows, adds cost, and RDS storage can’t be shrunk later.
- Good point on slot lag. 850/800 is a good move.
- Nice approach, much cleaner than maintaining Stripe IP allow lists. Once you confirm it’s working I’ll factor it into the billing service step for EKS: it’ll need a pod identity role with read access to SQS.
This is just from today.
https://redd.it/1x9umrb
@r_devops
optscale-kompressor: open-source proxy that compresses coding-agent tool output (−22.7% on real API bills)
https://preview.redd.it/vulcz3l6cdwh1.png?width=2540&format=png&auto=webp&s=09f90c56a3d90ce80a553eca942b12a250513a7b
Hello everyone! We open-sourced optscale-kompressor (MIT, single Rust binary).
If your team runs coding agents on API keys, a chunk of that bill is the agent re-sending the same apt-get output and compiler warnings every turn. In our traces that was about half the prompt. The compressor sits between the agent and the provider and strips the repetition, keeping errors and anything the agent can't reconstruct, and it can fetch back whatever it removed.
Everything is rules, nothing is learned, so the same output always compresses the same way. That matters more than it sounds: break the byte-for-byte prefix and you lose the prompt cache discount, which is how these compressors usually end up costing more than they save.
Real numbers off actual API bills, not token math: 22.7% cheaper on Terminal-Bench with Codex, solve rate unchanged. Works with Codex, Aider and opencode. Nothing phones home. https://github.com/hystax/optscale-kompressor — a star helps if it's useful to you.
https://redd.it/1x9pitg
@r_devops
Weekly Self Promotion Thread
Hey r/devops, welcome to our weekly self-promotion thread!
Feel free to use this thread to promote any projects, ideas, or any repos you're wanting to share. Please keep in mind that we ask you to stay friendly, civil, and adhere to the subreddit rules!
https://redd.it/1x9pgb3
@r_devops
Whats’s your experience with SST.dev or Pulumi?
I’m working on a personal project, so 0 users, and I have worked with SST in the past, but having to upgrade SST was a pain. Not sure if pulumi is any different of an experience, or is less painful and wanted insight on different experiences with either tool
https://redd.it/1x9js9y
@r_devops
Should I teach myself the AWS CLI or is the time for that is gone?
I'm setting up an AWS workflow (we work only with azure and on-prem at work so little experience with the CLI) and wondering if I should bother learning about the tool itself.
Seems like I should just set up limited users with limited roles, hard budget caps, place manual gates on production CIs and let a sandboxed AWS MCP handle everything else. Using Terraform is an implicit admission on how we'd all like to instrument our architecture in a vendor-agnostic vocabulary. An MCP seems to be an additional step in the same direction?
Thoughts?
https://redd.it/1x9aq1z
@r_devops
Hosting my own multiplayer game
Hi all! I wanted to share a project I've been working on.
In my free time I've been learning how servers work, especially how multiplayer games are handled on the server side. I built a small game and now I'd love for people to play it and push it as hard as they can. The more players and chaos, the better: I want real world logs to see how it holds up under load.
If you end up enjoying the game, let me know! I find it surprisingly fun myself, so I'm considering expanding it and improving the visuals down the line. That's not the main goal for now, though.
The game: https://airdogz.hobolichi.com
https://redd.it/1x8w1k0
@r_devops
How do I actually scan for CVEs for Javascript?
Hello friends.
I am implementing CI security scanning with Syft generating the SBOMs and uploading them to Dependency Track.
Two disclaimers:
1- I hate frontend work and have a poor understanding of it.
2- I am not a cybersecurity guy. This is just a new avenue I am exploring.
I've faced an issue currently with CVE scanning for Javascript files, specifically it seems that there is no way for me to actually tell what packages made it into the bundler vs what was included in the lockfile.
This leads to the JS sbom including all the dev dependencies with nothing actually telling me what made it into the bundle.
For SPAs, this can be mitigated by ignoring dev dependencies, however most of my stack is Laravel, and from what I understand Laravel puts all its NPM packages in devDependency, meaning I cannot skip them in this stack.
I have also tried generating sourcemaps while building (and I know you are supposed to .dockerignore them as it's a security concern), but without a reference to compare against, I do not know if this approach actually works or if it's a false sense of security. It does reduce the resulting sbom significantly, which seems to be a step in the right direction.
I've also read this study from 2022 https://das.encs.concordia.ca/pdf/Latendresse_ASE2022.pdf and it seems to reinforce my point that what I'm asking for is (at this time) not possible.
My goal is to understand 2 things:
1. Is it actually not currently possible to know what in the lockfile actually made it to through the bundler and into the final app.js that gets served?
2. If 1 is true, do I just not run CVE scans on the lockfiles for Laravel projects? Do I just scan everything and force people to upgrade as much as possible to just "lower CVE number"?
https://redd.it/1x8peyq
@r_devops
Looking for an affordable production-grade VPS/EC2 instance as a solo developer (4 vCPU, 8 GB RAM)
Hey everyone!
I'm a solo developer working on my own digital product, and I'm looking for an **affordable but reliable server for production deployment**. Since I'm managing the project and infrastructure myself, I want to keep hosting costs as low as possible without compromising too much on performance or reliability.
Here are my requirements:
* **CPU:** 4 vCPUs / cores
* **RAM:** 8 GB
* **Storage:** 50–60 GB SSD or NVMe
* **Usage:** Production workloads, not just local development or testing
* **Budget:** As economical as possible, preferably with predictable monthly pricing
I've been looking into AWS EC2, but I'm also open to other VPS providers if they offer better value for money. I'm not specifically looking for an AWS Free Tier instance; I'm interested in something I can realistically use to host a production application.
I'd appreciate your advice on a few things:
1. Which VPS or cloud provider offers the best price-to-performance ratio for these specifications?
2. Is there an affordable EC2 instance type that would be suitable, or would a provider like Hetzner, DigitalOcean, or another VPS host be a better choice?
3. Are there any genuinely free or heavily discounted instances that are suitable for production use, rather than just temporary testing?
4. Are there any hidden costs I should consider, such as storage, bandwidth, backups, IPv4 addresses, or taxes?
I'm happy to manage the server myself, including Linux configuration, deployments, and application maintenance. I mainly want to avoid paying enterprise-level prices when I'm running a product independently.
**If you're a solo developer hosting a real production application, I'd love to hear what you're using and how much you pay per month.** Personal experiences and recommendations would be especially helpful.
Thanks in advance!
https://redd.it/1x86u37
@r_devops
AI Identity Crisis
I'm a senior with loads of years, i've been doing devops/sre and systems engineering in the cloud for a long time. recently i volunteered for a project that i knew would lean heavily on claude, and with my manager's enthusiasm and support it looks like the project will be a success. aside from not truly owning that success, i'm also more than a little concerned that the 10K lines of code that i've slopped out, while mostly robust and functional, may not be personally defensible or supportable by me. this is a "career first feeling" for me, and i'm wondering if others have had similar experiences recently. is this just the way it is? maybe i should just treat it like "i started a new job and inherited a bunch of new libraries" when things come up? I'm not really sure how this is going to end.
https://redd.it/1xbpjnn
@r_devops
What’s the worst production failure your monitoring didn’t catch?
Curious how other developers handle cases where everything technically reports success — HTTP 200, Lambda completes, webhook delivered — but the intended operation never actually finishes.
For example, a payment succeeds but the customer’s subscription isn’t activated, or an API call succeeds but a downstream system never gets updated.
Have you encountered anything like this in production? How did you discover it, and what tools or processes do you use to catch these failures?
I’m especially interested in cases where traditional logs, metrics, or error alerts weren’t enough.
https://redd.it/1xbbpob
@r_devops
Too many internal tools are becoming hard to maintain
We've built tens of internal dashboards, tools, and admin utilities over time. Now, vibe-coded tools are becoming the norm for us, and it feels like a maintenance nightmare.
We have to constantly update and fix them and keep them working with our ever-changing data structure. Also, It's getting hard to track what tools exist, who uses them, and what’s active vs. abandoned.
How does your team handle this? Are there specific systems or governance models you use to maintain the internal tools?
https://redd.it/1xb0b5k
@r_devops
Homelab runner for Gitea
Hello guys,
I'm new to the devops world and I'm working on some projects to learn. I'm currently trying to automate the creation of VMs with Terraform with my local Proxmox.
As CI/CD and Git server I have everything on a Gitea local. I remember doing some CI/CD once on Github and it worked fine but apparently I need a runner in my home.
Should I create a VM or a Docker container? I have no idea on how to setup that thing and to connect it to my Gitea right after.
Thanks for the help
https://redd.it/1xatqay
@r_devops
Sonatype Nexus now requires mandatory telemetry gathering
https://help.sonatype.com/en/nexus-repository-telemetry.html
https://redd.it/1xaremm
@r_devops
Is anyone using Endor Labs products?
Doing a poc and wanted to know about the product from someone who is actually using it in production. How has your experience been and how effective it is in filtering security violations?
https://redd.it/1xa2e3q
@r_devops
Jenkins Alternative for Projects with Mixed VCS
I run a small game studio (~10 years, been in industry for 15, just to note background) and am looking at modernizing our build machine. Previously, we used Jenkins as it was something I was familiar with, but setting it up and maintaining new projects has always been more laborious than preferred. I have used it's older build style heavily, relying on command line scripts, as well as it's pipeline workflow which took much more effort to set up, though I liked it's flexibility and stage reporting.
Our specific challenges are that we primarily do service work, so we are often in a mix of version control systems across hosts: git (github, gitlab, gitea, butbucket, etc), Unity Version Control (previously PlasticSCM and what we use internally for projects), and once in a while, SVN. Jenkins was fairly VCS agnostic, which allowed for "quick" setups across systems. Projects are not hosted locally.
Our previous builder was spun down a few years ago and we haven't replaced it. Our deliveries are now built on dev machines (usually my own) using build scripts for consistency. Most project builds are relatively quick but I'd like to get things offloaded and automated once more to a dedicated build machine. The spinning down of the old machine was partly due to a security incident that affected another team using Jenkins that had me reconsider the risks of using it.
The system needs to be self hosted with local runners as we mostly build for closed platforms and our preference is to keep everything internal, when possible. The target platforms primarily require Windows machines for building and local access to development hardware. It's uncommon that we work on mobile projects but we have local mac mini machines for that when necessary. Builds and build artifacts can be quite large, so self hosting also lets us keep costs down by leveraging our local NAS setup.
We don't really need webhooks to the VCS for our setup; manually triggering builds from a web UI is all the team really needs (it allows non-devs to get development builds for testing on hardware) and the occasional cron job for weeklies. On the other side, pushing notifications when builds are complete along with their status is important. We use Discord for internal communication which is often overlooked by other CI tools that prioritize Teams and Slack.
I've read through a lot of older threads about Jenkins alternatives and it seems that most suggestions involve using the CI included with the host, which isn't entirely possible with our needs. I would like to use something secure (and therefore less dependent on community plugins than Jenkins is) that lets the VCS hookup be swapped out in some simple manner per project.
The conclusions I have reached at this point is that there really isn't a good, free (or cheap) solution outside of Jenkins as teams move away from dedicated build software to full software platforms that provide end-to-end services for hosting, project management, and CI/CD. I would very much like to be wrong about this and have somebody inform me of a modern FOSS Jenkins replacement out there.
https://redd.it/1wydwfj
@r_devops
Starting a Team
looking for people to form a team. could be any position in tech.
https://redd.it/1x9q44s
@r_devops
To monorepo or not?
My startup is in consumer electronics. The product is 1 device with 1-2 peripherals that come with it. Users connect to a mobile app with a backend that does data analysis on what the product measures.
ATM I have 4 repos: a flutter app and a FastAPI backend with sqlite. The embedded side uses PlatformIO+Arduino for the device/hub, and same stack for the peripehrals.
The DevOps is getting unwieldy. On the one side, I'd like everything to be in one place so agents can easily build up a context and CI/CD is not scattered. On the other hand, I hate working with massive project trees and loading up vscode would be a nightmare.
Thoughts? At work I use git submoduels but I was told that's not a monorepo.
https://redd.it/1x9q2cd
@r_devops
How to monitor Supabase using an OTEL Collector?
https://telflo.com/blog/monitoring-supabase-with-the-opentelemetry-collector
https://redd.it/1x9kck3
@r_devops
Your AI coding agent might be cheating to pass CI. Here's how to catch it!
I had a small bug in a bill-splitting app and ran the same fix task 10 times across three coding agents. Every single run changed the test instead of fixing the code.
The failing test, encodes a product requirement => split 100 cents three ways and every receipt shows 33, even though 1 cent goes unaccounted for.
The agents treated it as a bug and rewrote the test to match their code 😆
What I learned: an agent's "tests pass" is worthless if it's allowed to edit the tests. You have to check what it changed, not just whether it finished green.
So I built Skill Gate to catch issues like this in our Cursor or claude skills so you can evaluate them before sharing with the rest of your team to speed up their programming workflows.
You give it a skill and a few test cases, it runs the agent on them, and a decision model checks what the agent did, like "were all files under tests/ left unchanged?", not just the exit code.
Limitations (for now): you bring your own API keys and write your own test cases.
Would love to hear any thoughts or feedback from the community here!
https://www.skill-gate-ai.app/blog/ai-coding-agent-cheating-ci
https://redd.it/1x9ipmw
@r_devops
Thinking about moving from backend Go to SRE/DevOps
I’ve been working in software for about 15 years, mostly backend, and I specialize in Go. I also have around 3 years of DevOps/SRE experience, though that was about five years ago and in a small company.
Lately, my backend role is almost entirely prompting and reviewing AI-generated code. At my company, even people who aren’t engineers can open PRs.. Management thinks that "help" us, but thanks no thanks, 1k+LOC PR is not helping anyone... They want us to build some magical "harness" that will produce exactly the feature you want with all good practices embedded.
I worry that my job is heading toward purely supervising AI-driven workflows rather than writing and designing software.
I’ve been considering moving back toward SRE, platform, or DevOps work. Perhaps it's naive, but I think it's less exposed to the AI frienzy, as it's not directly part of product. I get that a lot of things get automated (incident response, dashboard) but that's kind of in the spirit of devops ?
I see it as more job secure and more interesting in this day and age, as you'll always need people to run your infra for ai workflows (right anakin ?)
Also part of the appeal is that these roles seem more available where I live, while remote backend jobs have been hard to find. Devops skills can be reused in a lot of company of various sizes.
Question for you: how has AI changed your day-to-day work? Is it making the work more interesting, or just shifting more of it toward reviewing generated configs and automation? How do you see it evolving or enduring ?
https://redd.it/1x8ygo6
@r_devops
How does your team track everything that's about to be retired in your cloud?
Infra engineer here (about 10 years across AWS, Azure and on-prem). In every environment I've worked in, something is always approaching end of life: AI model versions on Azure OpenAI and Bedrock, Kubernetes versions on EKS/AKS/GKE, Lambda and Functions runtimes, database engine versions.
Each cloud sends its own notices, but they tend to land with account or subscription owners, not the team that owns the workload. And unlike library versions, none of this shows up in Renovate or Dependabot.
Curious how your team handles it:
\- How do you find out? Native notices (Service Health, AWS Health, Advisor), vendor emails, a script, or someone just happens to notice?
\- How does the notice get to the team that owns the workload?
\- Do you catch any of it in IaC or CI, like pinned versions in Terraform or a pipeline check?
\- What's your worst "we missed it" story?
\- If you use a tool or spreadsheet for this, what's annoying about it?
Full disclosure: I've built a few small tools around cloud AI model and Kubernetes lifecycle data, and I'm exploring whether something for this specific problem is worth building. Mostly I want to know if it's a real pain or already handled well. I'll share a summary of what I learn.
https://redd.it/1x8ubmw
@r_devops
How does your organisation handle CI/CD and image tagging for microservices?
Hi everyone,
I’m trying to understand how different organisations design their CI/CD workflows for microservices, particularly the deployment process after a Docker image is pushed to a container registry like JFrog Artifactory/ECR
I have a few questions:
Image tagging strategy: Do you use Git commit SHA, build number, semantic versioning, or any other strategy? How do you promote the same image across QA, staging, and production?
Deployment workflow: Do you handle deployments entirely through Jenkins pipelines, or do you use a GitOps approach with Argo CD, where image tags are updated in a Git repository and Argo CD syncs the changes?
Helm and manifests: If you use Argo CD, how do you manage deployment manifests and Helm charts? Does Jenkins update the image tag in the Helm values file, or is this handled differently?
Progressive deployments: Do you use Argo Rollouts with Argo CD for canary or blue-green deployments, or do you manage rollout strategies directly through Kubernetes deployments?
End-to-end flow: How does the overall process work in your organisation, from code commit and image build to deployment in production?
I’m particularly interested in understanding the different approaches organisations take and the reasoning behind their choices.
Would appreciate it if you could share your workflow or experience. Thanks!
https://redd.it/1x8d4ea
@r_devops
Who else has a "no deploys on Friday" rule, and what's the story behind it?
Every team I've been on has the rule. No deploys on Friday. Nobody remembers who made it, but everyone remembers the incident that caused it.
DORA's research keeps finding that elite teams deploy on demand, multiple times a day, with change failure rates under 15%. The Friday rule was never about Friday. It's about not trusting your own pipeline.
So who broke the rule, and what happened?
https://redd.it/1x8b8ks
@r_devops
