86
Reddit DevOps. #devops Thanks @reddit2telegram and @r_channels
Started DevOps trainee role in a startup one month ago , any advice
I recently got the job in a startup which creates and manages infrastructure and pipelines for other companies (clients) , there is a lot to learn here as I am working directly under a senior Devops engineer, my salary is kind of competitive according to the region , I am getting overwhelmed by the work and working hours are 9-6 , How can I manage so that I don't make a mess of my life in starting of my career
https://redd.it/1g4q5o4
@r_devops
Is there an easy way to see which containers triggered an error without explicitly sending the errors to a logging service?
I have 25 containers constantly sending messages to one another and sometimes one of them gets an error, but I have no idea which container got it. Is there a way to listen for errors on every docker container and centralize logging without explicitly writing code to send error to a microservice? I am using a local docker environment.
https://redd.it/1g4ljef
@r_devops
I launched my DevOps PostgreSQL platform today - feedback?
My name is Elliott, I’ve been building a DevOps platform the last three years on the top best in class open source platforms (Kubernetes, Elixir, PostgreSQL, Grafana, etc). The goal is to give
engineering teams access to a modern DevOps infrastructure without needing to have full SRE/DevOps committed resourcing.
It’s also open source/fair source - all the source code is here → [https://github.com/batteries-included/batteries-included](https://github.com/batteries-included/batteries-included)
I just shipped a public beta today and would love to hear initial reactions, thoughts, feedback.
Here’s some of the specific details of the platform:
* The platform features a user-friendly suggestion-based interface that guides users on topics like PostgreSQL cluster memory/CPU ratios, serverless web hosting, and secure secret sharing. Advanced users can quickly access full control over their data.
* It’s an Elixir-based UI on a database-driven, self-hosted Kubernetes platform. It can automatically deploy a scalable cloud installation (currently on AWS, with more options to follow) without the need for YAML or Terraform configurations. Alternatively, it can set up a development instance using Kind and Docker or Podman, facilitating a smooth transition from local to production environments.
* The platform supports easy AI project hosting for various workloads. Use Ollama embedding models for text embedding, eliminating OpenAI costs and data leakage risks. With PGVector and Cloud Native PG for vector databases, you can achieve near-state-of-the-art performance without exposing your data to third-party APIs. Experiment with Jupyter Notebooks, featuring optional Nvidia Plugin batteries for no DevOps-required experimentation.
* Single Sign-On is streamlined via Keycloak, Istio Ingress, and OAuth Proxy, securely hosted on your machine or cloud account. We've simplified security with full mTLS, Istio, SSL generation, and automated routing with Let's Encrypt and Acme for HTTP2. Istio Ingress services are seamlessly configured down to the contents of config maps.
* Grafana and Victoria Metrics can be auto-configured with just a few clicks for easy installation.
Here’s also a look at the demo of the database deploy [https://www.youtube.com/watch?v=YbvkWja3VIQ](https://www.youtube.com/watch?v=YbvkWja3VIQ)
The platform follows all the best practices learned for configuring and running a maintainable system without Kubenete's GitOps pain.
If you want to check it out, here are links to docs, site, repo, and join:
* [https://www.batteriesincl.com/](https://www.batteriesincl.com/)
* [https://home.batteriesincl.com/signup](https://home.batteriesincl.com/signup)
* [https://github.com/batteries-included](https://github.com/batteries-included)
* [https://www.batteriesincl.com/docs](https://www.batteriesincl.com/docs)
https://redd.it/1g49t4q
@r_devops
Advice on new architecture. No more AWS ECS, Ansible to orchestrate docker on EC2 instead. Am I insane?
The problem
I currently use AWS ECS Fargate and ALB to serve my api and run my background workers. I don't like this setup because 1) for what we're getting out of it it's too expensive, 2) I can't truly replicate it locally¹, and 3) most importantly, squeezing in DB migrations into an ECS deployment has been quite painful. DB migrations specifically highlighted to me that ECS seems to be the wrong tool here.
So l'd like for some thoughts on if I'm thinking right about completely switching away from ECS and the tools I'm picking to do so.
Some context on the application first though. It's a B2B app, so sudden demand increase is unlikely. I will scale up slowly when needed, but I do want to have a plan for when that need comes. Zero downtime deployments are also not a requirement, a few minutes of downtime at night are fine.
My plan
A single EC2 instance as beefy as needed. It's got the app running in containers and the way to scale is to increase the number of containers. I'll use traefik as a reverse proxy in front of the api containers. I don't know what the appropriate load balancing algorithm would be here, but I didn't think this is that big of a deal either, right? As in, would the answer to this question affect the architecture I'm deciding on? Or can I just revisit this after I've implemented everything to better optimize the load balancer?
When code is merged to master, GitHub Actions will build a new app image, and then I'll use Ansible to automate the deployment in that EC2 instance.
I picked Ansible instead of writing a custom bash script because it seemed that I can use Ansible to declare what I want to happen in some sense, but I can still also imperatively write how it should be done. Is that correct?
This is the most vague area to me to be honest, so feedback here is greatly appreciated. I have never used Ansible before.
Another relevant point to mention here as well is that my custom bash script would be annoyingly stateful, which seemed too error prone. For example I have to check and ensure the state of the machine first, like spinning up traefik if it's not already running, and checking the db connection, etc. Ansible seemed like it had a good approach to this issue. It's a complex issue though, so I'm keeping my expectations low. Experience with things like this is appreciated.
I'm planning on using parameterized docker-compose files to configure the containers and set their network, static env vars, entrypoints, etc., in addition to passing in dynamic configuration (i.e. from AWS SSM) as env vars as well.
Running the entire app locally would simply be `docker-compose up -d` or I can even simulate a deployment process exactly by running the Ansible playbook targeting my local machine.
Here's the deployment logic if it's useful to know:
(I will use a GitHub Actions workflow to run the Ansible playbook directly into the instance to update it)
- Pull the new container image.
- Add a higher priority route in Traefik to point api traffic to a maintenance page.
- Wait 30 seconds for any ongoing requests to complete since Traefik doesn't support connection draining.
- Stop (not remove) old app containers.
- Run DB migration using new app image. Migration atomicity will be ensured by a few steps that are irrelevant here, but what's relevant to know is that it will be done using direct access to the PostgreSQL CLI.
- If db migration is successful, spin up containers for new app version, remove maintenance page and clean up old containers. Deployment is done!
- If db migration fails, start old app containers, remove maintenance page and send alerts.
As for observability, Prometheus would scrape the local collectors (cAdvisor, OTel, etc.) for logs and metrics.
When I need to scale up the api or the workers I'll add more containers, and when I reach hardware limits I can upgrade the instance type. Any gotchas I should know about
Ideas for creating Dead Man's Switch emailing system
Hey guys.
I am not sure if this is the right sub for this, but I feel like you all are my best bet.
Well I am looking to setup a system that basically functions as a Dead Man's Switch that will send out an email to my family members in case I pass away or something. I have seen services like deadmansswitch.net, but there are a few reasons why I am not using their service.
Basically, it would have to work such that the system sends you reminders by email every now and then, and you have to click on a link. If you don't click the link within a predetermined period, the system will trigger and send out a predefined email to your recipients.
I am not a hardcore DevOp like most of you guys, but I know some basic programming. What would be the easiest way to go about building a homemade solution like this?
https://redd.it/1g487tb
@r_devops
OS factory : time and load for such a project
Hello DevOps around the world! 👋
I'm starting a new position at a large firm that primarily uses GCP and Azure for IaaS. Due to the nature of the business, we have a lot of software that isn't compatible with PaaS, so we're stuck with a lot of manual installations.
My team currently installs OS/Agents/Middleware/Software directly on machines, which is a huge time-sink and slows down our time-to-market.
To address this, I want to propose an OS factory project to my management. While I haven't built one before, here's my initial thinking:
* **Core Technologies to be used:**
* **Packer:** For image creation and management.
* **Ansible:** For configuration management and provisioning within the images.
* **GitLab:** To manage our CI/CD pipelines for the entire process.
* **Phased Approach (i want to keep it simple, and build small victories ✌)**
* **Phase 1:** Build a pipeline to create and push patched OS images daily.
* **Phase 2:** Pre-install monitoring agents on the images.
* **Phase 3:** Pre-install security agents on the images.
* **Phase 4:** Work with the security team to apply CIS benchmarks for hardening.
**My questions for the community are:**
* **Existing Open Source Projects:** Are there any open-source projects that could help accelerate this effort?
* **Experience and Effort:** For those who have implemented OS factories:
* How long did it take, and what were the resource requirements (people, time)?
* What were your biggest challenges, and what advice would you give?
I believe this is a relevant topic for many DevOps professionals. Thanks in advance for your insights!
https://redd.it/1g43r72
@r_devops
1 Month Until I Start My First Full-Time DevOps Role – Any Advice?
Hey everyone,
I’ve been working in IT/Cloud/Security for around 4 years, and recently started taking on some DevOps responsibilities at my current job. I’m really happy to share that I’ve just landed a full-time DevOps Engineer role at a great company. My official start date is exactly one month from today.
I’ve offered to visit the office before my start date to better get to know the team and familiarize myself with the projects they have going on. This way, I can gauge where I stand and identify any areas I might need to catch up on.
I’d really appreciate any advice or suggestions on how to best prepare for my first day. This is a big opportunity that I’ve worked incredibly hard to achieve, and I want to make sure I hit the ground running.
Small story time.... just a year ago I was feeling pretty lost. I was out of work, unsure of my next steps, and burned out from my previous role. I even questioned whether I wanted to keep pursuing the engineering path. I decided to take a break, regroup, and commit myself to turning things around. I hit the books, worked on projects, kept my public GitHub active, and sent out around 10 job applications every day. After countless rejections, I finally got the “yes” I had been waiting for.
Thanks for reading, and I’d love to hear any thoughts or advice.
https://redd.it/1g4440l
@r_devops
podman upgrade possibly causing kube-apiserver high cpu?
I was working on some golang middleware tying into opentelemetry and saw the upgrade for podman and took it.
About 20 minutes later my laptop fan starts going like crazy. Come to find out kube-apiserver is maxing out a core on my laptop. I use kind, so I blow away the kind cluster and recreate and its good. Then 20 mins later it happens again.
I enabled audit logging on the apiserver and sure enough there are a ton of watch calls constantly. The sourceIP seems to be something within podman itself. I'll paste a sample line in the comments.
Anyone else seeing this? I'm wondering if podman is going wild on kube-apiserver and causing the spike.
So I spun up a vanilla control plane with kind with nothing loaded. Starts doing the same thing.
https://redd.it/1g1jxit
@r_devops
GitOps - one deploy config per service version
I want to hear your thoughts on a problem statement that is not broadly discussed.
Let's say I've got a project in which I need to deploy multiple versions of the same service (different clients with different rollout schedules). Let's say each version of the service needs a different deployment config (env vars, secrets, whatever).
I'm using ArgoCD do deploy this services dynamically. I've got an abstract service helm chart that I use to deploy different services by feeding different deployment configurations.
Now I'm adding another layer to this, different configuration per service version. I've been thinking about the cleanest and most usable way of storing this configuration and I've come up with multiple possibilities:
# Option 1 - Big file per service with a block of configuration per version
# Option 2 - One file with base line configuration + one file per version for any version specific config. Periodically, we would promote those version specific config to the baseline.
# Option 3 - Store deployment configuration in the service repo. Helps a lot with organization but if you want to change deployment configuration you need to rollout a new version of the service, which doesn't make sense.
# Option 4 - GitOps repo would contain a folder per file, inside a deployment config file per deployed version of said service. This one is the most understandable but the number of files could be exponential. (let's say you have 40/50/100 clients, each using different versions).
What do you think? How do you handle this? Do you handle this at all?
Hopefully as part of our ArgoCD/GitOps initiative we will be able to reduce the time between deployments and minimising this issue
https://redd.it/1g1cs7m
@r_devops
Best way to do CI/CD on a self-hosted server running Proxmox for a small web app
Hello!
I'd like to add CI/CD to my small web app that's on GitHub and hosted and is self-hosted. I'm not looking for the easiest (but rather the best) solution as I'd like to learn something new that might be useful to me in the future. This app is literally used by me and my collegues, so there's basically no traffic on it.
The app uses:
Next.js - Frontend
Python with Sanic - Backend
Postgres
Redis
Right now all this is in 3 separate LXC containers (API and Web are in the same one as the API is exposed thru Next.js rewrites). I did my research and it seems like the way to go is Portainer and a GitHub Action that builds a container and then pushes it to Portainer to deploy (So this solves CI too!).
My questions:
1. Is this a good solution?
2. Does it make sense to run all services related to the app in 1 Portainer instance (So that is the whole web app in one LXC basically with Postgres and Redis alongside it)?
3. Related to 3., if there was another web app, would it make sense to have another separate Portainer instance for it in another LXC?
Thank you!
https://redd.it/1g1dtr9
@r_devops
If you're struggling to learn, we have a bunch of projects!
Hey everyone, I'm Dan from roadmap.sh (which I know gets posted and mentioned all the time haha)!
We've been working hard on providing people with projects to help prove their knowledge, because as you know, the best way to really learn something is to build it!
We now have 21 DevOps projects that you can build with a good amount in Basic, Intermediate and Advanced!
https://roadmap.sh/projects?g=devops
If you want to see any other classic projects here then just submit an Issue on GitHub.
https://redd.it/1g19yc4
@r_devops
Naming conventions for VMs?
Hello,
New to DevOps. Just started this role less than a month ago.
I am being tasked currently with writing up terraform for the existing infrastructure that was created through the cloud provider WebGUI, and with that I’m being tasked with coming up with a naming convention for these instances since there isn’t really any consistency between them. I have to account for environment, and scale.
So- I’m thinking most general -> least general, so these instances are grouped alphabetically by their env essentially.
For example- dev-app-01 or something.
Do you guys have any recommendations? Any tips or advice?
https://redd.it/1g17eco
@r_devops
After 3 Years on the Same Tech Stack, What Skills Should I Refresh for DevOps?
I've been on a consulting project with a bank for the past three years, but now that it's wrapping up, I'll be on the bench. My work has primarily involved GCP migration from on-prem using GitHub Actions for CI/CD and Terraform Enterprise for IaC and deployments. After three years of sticking with the same tech stack and mostly writing YAML, I feel like I’ve lost my edge and need to refresh my skills. Any suggestions on areas, tools, or skills I should focus on to get back up to speed?
TL;DR: Spent 3 years on GCP migration using GitHub Actions and Terraform. Project’s ending, and I feel rusty. What should I focus on to stay sharp in DevOps?
https://redd.it/1g16hkx
@r_devops
Transition to Solution Architect?
I have a few years of experience in DevOps now.
I don't have a cloud cert under my belt just yet. Recently started working on getting AWS Solution Architect Associate cert.
I did a take home architecture exercise to get my current job. It was interesting and made me think, perhaps I would enjoy architecture more than DevOps. Maybe, I'm a big picture person?
DevOps is fine but I'm not sure I see myself doing this for the rest of my career.
How can I transition to a Solution Architect role? How would I know if being a Solution Architect is right for me?
Are there any Solution Architects out there that can tell me about their day-to-day?
https://redd.it/1g15168
@r_devops
How popular is this Wolfi base image as alternate to Alpine? Do you use it in production?
I am exploring ways to move away from Alpine as I encountered some DNS problem with it recently. Is Wolfi a good alternative base image? Please don't suggest bloated Debian and Ubuntu
https://redd.it/1g0wgky
@r_devops
How much of a challenge are telemetry (metrics, logs, traces) storage costs for your team / company?
Real-life cases of overpriced/inefficient used telemetry storage are very welcome in comments!
View Poll
https://redd.it/1g4iysh
@r_devops
Starting Devops (no cs background
Hey everyone I’m starting to learn devops buying a course on udemy by Imran teli I’m seeking for advice and suggestions about things while learning devops also will my no it/cs backround affect my hiring process once i’m ready to work??
https://redd.it/1g4k5xp
@r_devops
when upgrading instance types? From what I read in the docs, upgrading within the same family seems to be a simple task, no?
I know this is all over the place, there's a lot of things that need to fit in together properly, but I tried to only mention what's relevant. If there are any points I forgot to mention, I'll be happy to answer.
If what I'm asking for isn't clear I can also try and clarify that further.
Please mention any pitfalls I might fall into, even if you don't think they apply to my situation.
---
¹: I know that as some point replicating a scalable system locally is not a realistic expectation, but in our current state I see no reason why I can't spin up the same containers and reverse proxy locally and get the same deployed setup on my local machine. When the need comes for something like a hosted global service, I'll drop that requirement, but for now I don't see why I shouldn't be able to do so.
https://redd.it/1g4alqw
@r_devops
How common is it for companies that host hackathons to forbid contractors from participating?
Understand there are a variety of opinions on hackathons.
I work at a place that forbids full time contractors from participating in them. I'm trying to understand if the policy has a legal basis, if it's financially driven or has other motivations that aren't apparent to me.
https://redd.it/1g49sjx
@r_devops
DevOps online courses help
Hey everyone,
I'm coming from a JavaScript Full Stack background and looking to transition into DevOps. I found the "DevOps Beginners to Advanced with Projects" course on Udemy ( https://www.udemy.com/course/decodingdevops/?couponCode=2021PM20 ) and was wondering if it's a good starting point to be able to pursue a junior position in the field. Has anyone taken this course? Would you recommend it or suggest something else?
I was also recommended a more specific aws and cka courses but I'm aiming to accomplish them after going through a complete DevOps course.
Thanks in advance!
https://redd.it/1g440rd
@r_devops
What are your best tips for automation without over-complicating things?
I've been working on automating a few tasks in our DevOps workflow, but sometimes I'm adding unnecessary complexity rather than simplifying things. I'm curious to hear from the community:
* How do you decide what’s worth automating and what’s the better-left manual?
* What tools or techniques help you keep automation simple yet effective?
* Any lessons learned from over-complicating automation in the past?
Thanks in Advance!
https://redd.it/1g44mox
@r_devops
Connect Cloud Build and Bitbucket Cloud
Hey guys, devops newbie here. Currently I’m trying to find an alternative to Bitbucket Pipelines due to some limitations related to its self hosted runners (lack of build concurrency, build step timeout only 2 hours).
I am trying to see if Cloud Build is a viable alternative due to its private worker pool option (it’s managed as well).
My company’s repositories are hosted in a workspace on Bitbucket Cloud behind an IP Allowlist.
I’m having trouble trying to connect Cloud Build to a repository since during the link repository process GCP uses an external IP address from a range it has allocated for itself. Google publishes the allocated ranges in a webpage as a json and updates them frequently.
However adding these ranges to our IP Allowlist cannot be a safe choice since this is a public IP range.
Before I move on to another CICD solution, is there something I’m missing to make Cloud Build work?
Please let me know if I need to provide more information.
https://redd.it/1g425zj
@r_devops
Does it make sense to use trunk based development with canary deployments?
I've been reading a lot about deployment strategies recently to decide what to go with for my early stage startup. Stability is important as it's a website for lawyers to run their practices.
I want to do trunk based development, but wondering if it makes sense to pair it with canary deployments?
Say upon a release to production, for the next 24 hours, only 10% of users are routed to the new version. Given that no issues are caught over that period, all users are then routed to the new version.
The benefit is that any issues caught in production will affect only a small portion of users.
The drawback is some (minor?) complexity in setting this up with gcp cloud deploy, as well as monitoring canaries for every release.
Should I implement canary relases?
https://redd.it/1g1gr6s
@r_devops
Dashboard for Apache with Geo Location based IP address
Hi all,
Please suggest an dash board ( Prometheus + Grafana ) for Apache with Geo Location map based IP address.
https://redd.it/1g1eq15
@r_devops
I am stuck in my job don't know whether to quit or go with the flow?
Please need genuine advice
Currently I work as the L1 NOC engineer and my work includes Linux OS, Networking, Putty, NS-OX, and communication with customers to resolve issue.
Now The scenario is earlier I was doing an internship in the startup based company and the role was Frontend dev. I left that internship because of this job due to higher package and the HR told me that they have various fields in the company so they will put me in web dev and I accepted the offer but later they put me in this NOC position and told me after 6 7 months I will get the domain of Devops, Cybersecurity, Cloud, Network, Database, and Backup. I don't trust them because there are many other people waiting for domain who are hired with me so it's gonna be in the randomised order.
Now my major concern is what to do here should I start studying for Devops and build projects in that to get a internship or entry level job which is quite difficult because no one hires a freshers devops engineer unless you are lucky. Or I should grind my Frontend skills and work on the js frameworks to get back in the web development field. Because I only Know HTML, CSS, JS and some react concept.
Currently its my fourth month here and there is nothing new to learn here and it's feel like this experience is nothing but just a waste of my time but the experience letter would say IT Operations Associate.
https://redd.it/1g1cble
@r_devops
Cloud and devops vs ml
Currently in 3rd sem been doing web dev for 7 months , I am not that good in web dev as of now but for long run I am thinking to do cloud and devops after web dev
don't have any prior knowledge of ml so it would be totally new as for future what should be my goal to learn after web dev should it be cloud or ml
I don't have a clear goal as to what to do I am just learning tech stacks and all and am bored doing web dev so thinking of switching to something else
https://redd.it/1g172ct
@r_devops
What project DevOps can build to make USA peoples say "Wow wow wee waa!"?
Jagshemash, DevOps neighbours!
It is I, Boyan, greatest DevOps in all Kazakhstan! I come to you with important question. I want to show my skill to USA companies—yes, land of McDonald’s, Pamela Anderson, and big monies! But how can I make them say, “Wow wow wee waa! This Boyan, we must hire him immediately!”?
What project can I make as DevOps engineer that is big and glorious? Something that will showcase all my big brain powers and make US and A recruiterka slide into my DM like smooth homemade rakiya.
Here is what I know to do very nice:
Make pipelines go fast, like rocket on cow’s milk.
I best snake handler in village: mostly pythons.
Automate things so I can rest and eat more cheese while servers run themselves.
Kubernetes? Yes, I can do! Even my neighbor Nursultan say, “Boyan, you are kuber-whatever genius!”
I also do monitoring, alerting, and can fix everything with only 3 lines of code—maximum!
So what can I build? Maybe I make:
Big project with CI/CD pipeline that deploy faster than gypsy stealing chicken?
Or I make kubernetes cluster that self-heal like strong Kazakh man?
Or maybe cloud infrastructure that so big and scalable, it can hold all of Kazakhstan’s goats?
What will make hot recruiter lady say, “This Boyan, we need him on remote contracts, fast!”? Please help me, friends! I want to bring my glorious DevOps talent to America!
Chenquieh!
Boyan Balgaran, soon-to-be American DevOps superstar
https://redd.it/1g173ww
@r_devops
A Self-Hosted Code Review and Analysis Server
We have built a self-hosted code review service, designed to be useful in the following scenarios:
* You have many repos but still want tight control over code quality
* Your repos are private, and commercial services seem overkill
* You want to continuously improve the process and rules, with full customization
We are open-sourcing it and hope it will be helpful.
[https://github.com/qiniu/reviewbot](https://github.com/qiniu/reviewbot)
Welcome feedback and suggestions. Thanks\~
https://redd.it/1g15h3s
@r_devops
Advice for new manager of a small team (3)
Hi - new manager here.
My background is SQL, python, powerBI automating data collection, creating reports and dashboards.
I used to work solo, on an island, with no real experience maintaining code for anyone but myself.
I’ve got my team a repo and we are working to deploy a pipeline and I need help with how we are going to manage our branch / branches…
Right now our plan is to spin off new branches for work items and push them into main when they’re ready.
We meet Monday + Thursday to chat formally.
I have set up coding guidelines and a repo.
What else can I do?
https://redd.it/1g0zrqo
@r_devops
how to manage secrets in gcp to have stateless projects?
Hi,
I have a shared project called tooling and several environment projects: dev, staging, prod
My idea is to have all dev,staging,prod stateless
But then how should I handle the secrets? we will be using mostly GKE for the apps.
I am using Terraform CDK with typescript.
My ideas are:
1. in Github actions store the secrets and in the app deployments create gke secrets replacing the map in every deployment so it does create/updates all the time and in one operation.
2. in Github actions store the secrets , create the secrets in Google secret manager at the app deployments, the problem is to handle the initial value and the updates, check if exists then create else create one version every time the app deploys? is not too many versions?
3. keep all secrets in the shared tooling project, for all the environments?
Any ideas? thanks
https://redd.it/1g0x658
@r_devops