86
Reddit DevOps. #devops Thanks @reddit2telegram and @r_channels
automate VPS server migration - what do you use?
im tired of doing it by hand - so I got ansible setup and can provision new hosts quickly.
But what do you do for data/application migrations old vps to new vps?
is it possible to automate that as well? using what tool? just rsync or is there anything better?
https://redd.it/1g85z0g
@r_devops
Buildfarm on Kubernetes
Anyone running production-grade Bazel Buildfarm (I guess it's just Buildfarm now) setup using the official helm chart with Linux, Windows and macOS workers? Just curious about experience with it, especially autoscaling, CAS management, overall configuration etc.
I have a setup with all that, but without autoscaling (because when Linux workers scale down buildfarm still looks for what was in their CAS for some reason and builds fail when they don't find something that was cached - instead of just rebuilding it, for some reason). Windows workers performance is kinda dogshit, but that may be about our custom toolchains - or just Windows being Windows keeping that CPU usage under 40% pretty much always.
As we're on AWS / EKS, we're also thinking about moving CAS to S3. Anyone here had something like that?
https://redd.it/1g84lz5
@r_devops
shared developer setup for macOS and windows
Hello,
recently we have finally been able to use MacBooks in our company, i.e. also for our customer projects.
Previously, development was done purely on Windows laptops and there was a setup with SEU-as-Code that created an isolated environment with IntelliJ, Git for Windows, VirtualBox for a Devbox and SQLPlus.
The environment was then assembled as required using configs, start and env scripts.
However, the whole setup is very old and should have been modernized a long time ago anyway, as SEU-as-code has not been developed further for years. Now they want a new setup and support for both operating systems
The current statement from our leads is that the costs for the modernization will increase by 50% if we want to support Mac and that they would like to use separate setups.
However, I find it hard to imagine that we are the first to run into this issue and that there is no practical solution, especially if the setup is developed from scratch and no technical debt has to be taken into account. Also, both leads were not fans of the Mac launch before, so I'm not sure if this is a purely objective opinion.
The question here in the round whether someone already has experience with this and perhaps has something like this in use in their own company.
Ideas, tools, anything would help, as I'm currently worried that our bosses might scupper the whole issue of Macs because of the statement.
Or does anyone know a better subreddit for the question if I'm wrong here, I haven't found one so I'm asking here in the hope it's OK.
https://redd.it/1g7x9vk
@r_devops
Memory waste and cold start
Drowning in config files and CloudWatch logs. Two issues I was thinking about sharing with you and gathering your thoughts:
1. Memory Waste: A study found 95% of serverless functions use <10% of allocated memory. How are you optimizing this in your CI/CD pipelines?
2. Cold Starts: Balancing performance vs. cost is tricky. What's your strategy for managing this, especially for low-traffic functions?
I'm curious:
* What tools or custom scripts are you using for serverless optimization?
* How are you handling observability and cost management for serverless in your DevOps workflows?
* For those at scale: Any tips on maintaining CI/CD speed and configuration management as your serverless footprint grows?
https://redd.it/1g7ubcm
@r_devops
Is it good to learn DevOps through openstack
Hi folks
I have 6 nodes and i wanna learn DevOps through openstack
What are you opinions about this
https://redd.it/1g7i7iw
@r_devops
How to authenticate paid software?
Context:
I have a freemiun app, most features relies on a local LAN server, except for a proxy server and in app features for the admin of said server.
What I've been thinking..
Method 1
- Being online most of the time to ensure the user is using a valid key, but this creates a conflict with the core of the app (minimal use of the network)
Method 2
- Ship the app with a public key to validate the user key, this key has encrypted data of the user and expiration date, this is stored on the client's device. Upon key expiration it's invalidated and removed from the device. The user has to pay again. (This one only relies on the network once per key activation)
What other methods can you think of for this context?
https://redd.it/1g7hg5k
@r_devops
Automate Deployment config changes?
There is something I have always been wondering about how to best solve this. The problem:
Deployments in Kubernetes cluster based on helm and ArgoCD. Now most things can be automated quite easily with this setup but what always seems to become troublesome in bigger projects are changes to configmaps and secrets especially staging these when they are environment specific.
Current setup:
Developers try to document all required changes and set values in a secret store that is referenced. This however still requires a lot of effort before deployments to change some environment variables in helm charts and secret references etc.
Is there a setup to fully automate this easily? We have a ton of different staging environments >25...
Edit: All generic environment variables and configmaps get baked into the helm base charts/images already
https://redd.it/1g76zcp
@r_devops
How Meta Uses LLMs to Improve Incident Response
I came across this interesting blog post on HN outlining how Meta uses LLMs to get 42% accuracy in root causing incidents: https://www.tryparity.com/blog/how-meta-uses-llms-to-improve-incident-response
https://redd.it/1g6q0fi
@r_devops
The new release of Dockerfile.app has launched.
Visit https://www.dockerfile.app
Features:
→ Save dockerfiles
→ Browse them
→ Upvote them
→ Search for dockerfiles
→ Create an account
All to create a community-driven location to get top-notch dockerfiles for all languages and frameworks.
Bugs? Let me know.
Feedback is welcome.
https://redd.it/1g6m2ce
@r_devops
Promoted to Manager
I've been in the sysadmin space for 7ish years, plus another 3 as DevOps engineer. The change into DevOps at first was a bit rocky and I think I suffered from imposter syndrome. I was the weakest one on the team (in terms of hard technical background and devops "years"). In recent months I realize I actually did beat out my coworkers at other skills: communication, organization/planning, enabling/empowering other engineers, not letting "great be the enemy of good", making actual progress. Recently we had a bit of a reorg and now I am managing the DevOps team. In a way it makes sense to me, I know the principles and our goals and the bite-sized chunks it takes to get there. I'll never be able to write slick bash one-liners on the fly with 5 ppl in a zoom meeting watching. Sorry for the rant and tamble. Long story short: any tips or suggestions for this transition from engineer to managers? Do you think I have enough background to succeed? Any suggested material or reading? (rn I've been reading Radical Candor)
https://redd.it/1g6hdmb
@r_devops
How can a devops engineer develop in machine learning?
Hi everyone, I'm a Devops engineer and I really love my job. I love doing linux, writing scripts, configuring networks and so on. But I also love math and algorithms and data structures. I want to participate in the development of artificial intelligence, apply my math knowledge, but still keep doing devops. Any advice?
https://redd.it/1g6gacx
@r_devops
How Well Does "all in the same repo" CDK approach Scale?
I asked this in the r/aws (here) but will also post on here.
I am in the process of adopting and learning about CDK for our large-scale microservices architecture. What I want to know is how well does it scale when used in an environment with 100s+ of microservices and pipelines.
Has anyone got any recommendations on best practices in terms of structuring and managing CDK for scale? Does anyone have experience using CDK in environments with 100+ microservices?
I can see that the biggest shift with CDK is essentially coupling the CI/CD config, infra config and application code all in the same repo. How does this approach/recommendation scale?
Let's say I have 100s of microservices and I need to update CI/CD or some infra config across all. Every time you make a change to the pipeline config in the repo, you are potentially "touching" the app and making a release. I can accept the changes to the infra "close" to the app like Lambda config, SQS etc., but I'm not sure CI/CD config is the same.
How do others manage updates to shared infrastructure or CI/CD configurations across multiple services?
Also, regarding self-mutating pipelines: it's something I tried 5 years ago with raw CloudFormation but found that if there was a change to the CodePipeline executing the change to itself, the execution would instantly fail and I would need to rerun it. Has this been fixed?
Lastly, why would a developer want to see the "pipeline update" step execute and do nothing 99% of the time, just wasting time and slowing down the CI/CD cycle?
I'd love to hear about your experiences and best practices for using CDK at scale. Any insights would be greatly appreciated!
https://redd.it/1g6dfye
@r_devops
First time creating a simple CI/CD pipeline help
Hey guys,
I'm a self learning developer trying to broaden my skills and learn more about cloud services and CI/CD pipelines and all that. To start simple I want to host my personal website on an EC2 instance and set up github actions to containerize and deploy to my instance such that when I push changes to my repository, that all happens automatically. My first basic question is how do you handle private environment variables? Do you use AWS parameter store? I'm using Go and the 'godotenv' package, but if I change my code around to use the AWS Parameter store, how would I test it locally? Since I wouldn't have access to the parameter store, right?
Thanks!
(you'll probably see me a lot around here in the near future 🙌)
https://redd.it/1g6auuk
@r_devops
Move from Sprints to KanBan?
For a long time I have hated sprints and pretend agile for DevOps in our organisation. We plan, refine, retro, demo - blah blah blah - the whole thing in our org is pointless because we plan/refine but when the sprint starts we abandon the sprint for another team’s work because they didn’t think to involve us early enough or they say no when we asked if they had any requirements, and now we’ve blockers.
Instead, we’re going to use KanBan. Anything in to-do can be picked up by anyone at anytime. Items only move from backlog to to-do after it has been refined, backlog items don’t show on the board, and board is DevOps only work. We no longer include other teams work on our KanBan board.
I’m not saying we won’t do work for other team. Instead we rotate through other teams. Embedding in their process and our tasks in their epics belong on their board because they belong to that team.
My fear is - how will be prioritise our own board if we’re embedding in other teams. What happens to unfinished work after a rotation.
What are peoples thoughts? Is there a better approach?
https://redd.it/1g61gzq
@r_devops
Manage spot nodepool in GKE
Hi Everyone,
We run about 60-65% of our workloads on spot VMs, but during peak hours we usually hit stock out and new pods are usually in pending state for long hours waiting for a spot VM. So we have implemented 2 ways to improve this state.
1. deploy the same deployment on payg nodepool with a higher hpa threshold, so it scales when spot pods doesnt scale.
2. create a nodepool with different series of machines with same configurations, taints and labels, but at times one nodepool doesnt scale even if it isnt hitting stock out, whereas the other nodepool would have stocked out.
Are there any better ways you guys tackle the stock out situation ? Kindly advice.
Thanks !
https://redd.it/1g5wxze
@r_devops
Software Engineer Jobs Report 10/16: Every week I spend hours scraping the internet for recently posted software engineer jobs. I hand pick the best ones, put them in a list, and share them to help your job search. Here is this weeks spreadsheet. 260+ roles USA and aboard. Devops roles included.
Hey friends, every week I search the internet for software engineer jobs that have been recently posted on a company's career page. I collect the jobs, put them in a spreadsheet, and share them with anyone whose looking for their next role. All for free.
I hand pick the ones I know are good roles, with market salaries, and no glaring flags (ex: I generally only put roles with posted salary bands). Though its not easy to tell if the roles require leetcode or not. I want to figure out how to get the information in the future.
The data is sourced by my own web scraping bots, paid sources, free sources, VC sites, and the typical job board sites. I spend an ungodly amount on the web so you don't have too!
About me, I am a senior software engineer with a decade of work history, and ample job searching experience to know that its a long game and its a numbers game.
If there are other roles you'd like to see, let me know in the comments.
To get the nicely formatted spreadsheet, click here.
If you want to read my write up, click here.
if you want to get these in an email, click here.
Cheers!
https://redd.it/1g85r21
@r_devops
I wanna build my own cloud
Hi folks
I wanna build my own cloud that i wish to extend
Which approach do i need to go with and why
https://redd.it/1g83oes
@r_devops
I can’t write the right vagrantfile for hyperv
I want to learn ansible but i can't start because i couldn't make the vagrantfile to run well on hypery (i can't use virtualbox) so if someone can help me adjust a vagrantfile so it can run on hypery with the same vms
Please send me a message if you can help and i will really appreciate it
https://redd.it/1g7vnr7
@r_devops
Suggestions on what to improve on in order to take bigger rolesband advance career.
I have been at my current place for 6 years and it's been a team of 3 managing a multimillion infrastructure spend. I came on to migrate us from on prem to azure and I had full control of what we use. I recruited another engineer I knew at my last job and we have run a really stable environment over the last few years with the third guy that was hired from within the company.
We use terraform, cinc, PowerShell, and jenkins to manage all infrastructure and automation. I am extremely strong in azure and PowerShell. I would say fairly strong in the other two, and I have a little experience with some other things..
We are a msft shop and everything is running on VMs but we've got some things in Linux and we're integrated in some azure services like azure files, redis, blob, and networking services like virtual wan.
We were moved to GCP after we got bought and its ok, but I'm getting bored. I like to do infrastructure architecture and environment build outs but its just maintaining a lot of it now.
Are there industry trends/new technologies that I should be learning about to further my career for other possible jobs? I was thinking learn some more on containerization but I'm not sure. I've done cloud infrastructure and automation for about 10 years now.
Any input is appreciated.
https://redd.it/1g7mwib
@r_devops
Terraform Test or Terratest
We currently don't have any sort of unit tests for our modules and I was checking into both of these tools. They both have their pros and cons, but I wasn't sure what people thought about the pros and cons of each at this point. I picked up enough Go to play around with Terratest and I do like how easy it is to import a module and do basic things that I'd want to do. Of course the native Terraform testing framework uses HCL so that's a big plus, but I don't feel like there are many examples out there comparatively and that it would be harder to achieve certain tests not having the flexibility of a programming language like Go. Curious to hear what you think.
https://redd.it/1g7iyeh
@r_devops
Seeking Career Progression Advice
Hey everyone,
I'm looking for some guidance on my career progression and would really appreciate your insights.
I started off as a DevOps engineer focused on containers, cicd, automation scripts, etc. Over time, my career led me more towards AWS, and I ended up obtaining several AWS certifications (8). This compounded my interest and skills in AWS, and I guess you could say I'm a cloud engineer now. I still support applications through cicd and containerizing, etc, but that's not the highlight of my skills.
Lately, I've been feeling like I've been stuck in the same role or rut for too long. I considered moving towards DevSecOps or application security, but that seems to be along the same track I started off in (containers and pipelines).
I'm curious about what career progression paths I can explore now. Given my background, what roles or areas might be a good fit? Has anyone else been in a similar situation and made a transition that revitalized their career?
Thanks in advance for sharing your thoughts
https://redd.it/1g7a9ep
@r_devops
Seeking Some Words of Wisdom
Hi all,
I’m currently working as a Platform Engineer at a large multinational company, but my journey here has been anything but straightforward. I started my career 11 years ago as a .NET developer. After a few years, I began feeling stagnant and found myself drawn to the world of cloud technologies. Driven by this passion, I started teaching myself everything I could from online tutorials and guides, determined to gain the necessary skills in platform engineering.
About 3-4 years ago, I took the leap and fully transitioned into the Platform Engineering space, and, I’m happy with that decision.
However, I’m constantly reminded of just how fast the world of DevOps evolves—especially with the rise of GenAI, MLOps, and other emerging technologies. It’s exciting, but also overwhelming at times.
No matter how much I learn or how many projects I work on, I can’t shake the feeling that it’s never enough. I struggle with the question of whether I’m truly “qualified” to call myself a Platform Engineer. I don’t hold any formal Kubernetes or cloud certifications, but I’ve gained hands-on experience working with these technologies. Still, the lingering doubt remains—how much is enough?
I find myself feeling uncertain about areas like networking and Linux, especially since I transitioned from a purely Windows-focused background. This sense of not knowing enough sometimes makes me question my place in this field.
I’m hoping to hear from others who may have faced similar feelings or have advice on how to navigate these challenges. How do you balance continuous learning with feeling confident in what you already know? How do you define “enough” in a field that never stops changing?
https://redd.it/1g7684l
@r_devops
Are there any DevOps or Infrastructure-type jobs where you work on boats?
Weird Q. But any need for these roles on oil rigs or cruise ships or anything similar? I had a random thought that it might be a fun alternative / break out of normal society for a while. But no idea if there is any demand for these kind of jobs in those environments. Presumably basic IT support, but not sure what else.
https://redd.it/1g6mlxi
@r_devops
Books for experienced DevOps?
Hey
I would like to hear recommendations about books that go beyond explaining what is DevOps from zero
I'm thinking about The phoenix project as I heard its great for both beginner and experienced engineers in IT, and also heard about SRE Orielly book
What you guys think?
https://redd.it/1g6kon6
@r_devops
My employer is offering me a 65% raise and a bonus in the next pay cycle if I rescind my 2 weeks notice.
In the past year working in a start up, I had made a transition working as a senior cloud infrastructure engineer to a junior and now mid level full stack engineer. 2 senior cloud guys and 1 senior full stack engineer decided to leave our company to take roles in FAANGs (who also happen to be our customers for our product) these last few months. Although we re’orgd and some duties got divvied out amongst us. I got bombarded doing my job and taking on cloud duties again. My mental health has been killing me with deadlines, and management asking us to push new releases on a Friday, which takes up some of my weekend. I’m just so done. I been offered employment elsewhere and put my notice in so I can take a month off for vacation and reset. Well I got a call almost instantly from the CTO, Product, and CEO about anything they can do to keep me including offering me a promotion to senior, a huge raise, focus on backend development only, and a $25k retention bonus on the next pay cycle. The raise is about 10% more than the new employee is offering.
They want to give me the weekend to think over it. I’m contemplating on whether I should take the offer or not.
https://redd.it/1g6he1w
@r_devops
DevOps/Cloud Engineers: How Do You Manage and Visualize Your Infrastructure?
Hey everyone! 👋
I’m working on a new tool concept to help DevOps teams and cloud engineers better manage their infrastructure. Before diving deeper into development, I’d love to get some feedback from the experts—that’s you! 😊
If you have a few minutes, could you help me by answering some quick questions?
# 1. Visualizing Cloud Infrastructure
How do you currently visualize your infrastructure (e.g., cloud resources, clusters, VMs)?
Do you rely on built-in cloud tools, open-source solutions, or paid platforms?
# 2. Managing Costs and Outdated Components
How do you track cloud costs across different resources?
Do you encounter issues with outdated Terraform modules, providers, or infrastructure drift? If so, how do you handle them?
# 3. What’s Your Biggest Pain Point?
What’s your biggest challenge in managing infrastructure or optimizing cloud costs?
If a tool existed that could simplify visualization and provide actionable cost insights, would that be valuable to you?
Thanks a ton in advance! Your insights will help us make something that really solves problems for the community. 🙌
https://redd.it/1g6dazv
@r_devops
Interview experience
I recently interviewed in mid level company where they rejected me in second round stating that I do have experience in CI/CD tool like Jenkins , harness but i don't have experience in azure cicd. Why can't they understand it's just a tool, logic and workflow is same everywhere.
https://redd.it/1g6b7yf
@r_devops
What happens if I have multiple IP addresses in a single weighted routing record in route 53?
Basically the title.
I am in the process of migrating from simple routing to weighted routing and wanted to test using a few servers.
Currently, we have a single A record which is simple routing, it consists of all the server IPs.
I am trying to take out some servers and add some weighted routing entries for the same.
If I have 3 records,
Record A - weighted, 2 IPs, weight 50
Record B - weighted, 1 IP, weight 50
Will each of the IPs in record A get equal traffic, I.e 25%?
I was not able to replicate the above.
Please help.
Thanks in advance.
https://redd.it/1g67daj
@r_devops
How do you handle urgent communication when a team leader from another team doesn't respond?
Hey all,
I’m curious about how you handle situations where you need an urgent response from a team leader in another department or team, but they take hours to reply. For example, if you're waiting for critical input and it's been over two hours with no answer—how do you proceed?
Do you escalate the issue, follow up with other team members, or try to find alternative solutions? How do you balance urgency without coming across as too pushy?
https://redd.it/1g5z5y6
@r_devops
Should I get into devops?
I am a frontend software engineer and I hate my life, I am sick and tired of putting buttons, input boxes, etc. on a web app... And I feel like frontend web development is a dead end job, or at-least it is not suitable for me, because all the frameworks do the same thing... I hate most of my coworkers (including developers, design, product managers, etc.) because they fight over trivial issues like, why use a forEach instead of reduce? Why is this button small, etc.
Should I switch to devops? I feel like there is much more variety of tasks available in this space, but I am afraid of AI, platforms like vercel, that abstracts all the complexity of managing infra, etc. Also currently my salary is quite high, so changing role will be extremely difficult...
Is Devops worth it? Or are all the IT jobs the same?
https://redd.it/1g5uf5x
@r_devops