86
Reddit DevOps. #devops Thanks @reddit2telegram and @r_channels
Fraud detection data pipeline (ETL) on GCP
# How to create a scalable fraud detection steaming data pipeline
[rasvihostings/fraud-detection-data-pipeline-etl-on-gcp-2b15b8f3d65b" rel="nofollow">https://medium.com/@rasvihostings/fraud-detection-data-pipeline-etl-on-gcp-2b15b8f3d65b](rasvihostings/fraud-detection-data-pipeline-etl-on-gcp-2b15b8f3d65b" rel="nofollow">https://medium.com/@rasvihostings/fraud-detection-data-pipeline-etl-on-gcp-2b15b8f3d65b)
When we think about large volume streaming data pipeline three things come to our mind
* **Scalability and resilience**
* **Cost**
* **Infrastructure maintenance**
I designed a solution which can scale easily, use much as possible GCP managed services and finally reducing the cloud cost 😉
[read my blog post here](rasvihostings/fraud-detection-data-pipeline-etl-on-gcp-2b15b8f3d65b" rel="nofollow">https://medium.com/@rasvihostings/fraud-detection-data-pipeline-etl-on-gcp-2b15b8f3d65b)
let me know what type of articles you guys are more interested on
https://redd.it/1g5t04r
@r_devops
API logging recommendations
Hi, I'm working on a project for a foodservice company mainly focused on catering. We have a platform that allows customers to order catering from our in-house brands, but we need a logging mechanism for the site. The traffic will be relatively low especially given the nature of catering (only takes 1 user to order for many people), but as the company grows there's expected to be more traffic. I'm looking for a cost-effective solution for this. I've heard datadog is really good but I also hear it's expensive. Our software isn't extremely sophisticated and doesn't rely on "mission critical" data so I wasn't sure if that'd be overkill. New relic was suggested to me, but it's hard to get a feel on pricing giving the unknowns about data ingestion.
https://redd.it/1g5rb6i
@r_devops
Self-hosted GitHub Actions runners on Kubernetes with MicroK8s
Got a powerful bare-metal server somewhere? Could be a good idea to setup self-hosted runners there with a simple Kubernetes distribution like MicroK8s.
In this article I setup ARC with the officially supporter runner-scale-set version. It's actually not that hard, although there are some limitations.
https://redd.it/1g5ml0h
@r_devops
Is there an app or phone feature that would automatically notify clients of service delays after a certain time period has elapsed?
Hopefully, this is the correct reddit community to address this question. If not, please kindly direct me to a more appropriate one. I work for a home health care organization. The field staff often fail to notify clients via the company cellphone if and when they are running behind schedule. I'm under the impression this is due to the nature of their job. The job requires them to conduct hygiene and grooming services to vulnerable clients like the elderly or those with disabilities. There seems to be two main obstacles for the field staff. One, unexpected delays occur while staff are servicing clients, and due the clients risk for falls/ injury the staff members can not leave the client unattended and inform the office or other clients of the delay within a timely manner. Two, notifying a list of 10-15 or more clients each time a delay occurs due to unexpected circumstances isn't feasible due to the amount of time it takes to reach and communicate with the client as many of them have difficulty with mobility, hearing and the like.
Is there an app or phone feature that could send an automated message that notifies the next client and/or list of clients of a delay using a countdown? For example, x client has one hour time slot, and if the worker is still in the home past that time it sends an automated message to the next client of a possible delay.
https://redd.it/1g5l5sm
@r_devops
Order matters - making a compound index 50x faster
https://jaywhy13.hashnode.dev/order-matters-making-a-compound-index-50x-faster
https://redd.it/1g5gnai
@r_devops
Got my CKA and CKS this year! What’s next – Ansible or Istio/Cilium?
Hey folks,
I recently wrapped up the CKA and CKS, and preparing for both was incredibly rewarding—not just for the certs but also for the confidence boost. I feel like I’ve got a solid handle on Kubernetes now.
I’m looking to keep the momentum going but torn between two paths:
Red Hat certs (e.g., Ansible): To dive deeper into automation and platform engineering. It seems like this could unlock some cool DevOps/infra opportunities.
Istio or Cilium: To specialize in service meshes, networking, and advanced microservices management within K8s—feels like a natural next step for more specialized roles.
For those who’ve been down this path, what made the biggest impact for you? Did these certs help you land better roles or more exciting projects?
Appreciate any insights!
https://redd.it/1g5cwhu
@r_devops
Kodekloud standard or pro?
I'm thinking to subscribe to kodekloud for a year but I'm unsure if going for pro is worth it over the standard. The pro version gives you the pro courses, which there's not many of tbh, 60+ devops playgrounds and 3 cloud playground. Standard doesn't.
Anyone who's used KK. Was it worth going for pro or was standard sufficient?
https://redd.it/1g5ai2m
@r_devops
Dynamic DevOps Roadmap repo reached 1K starts
In less than a year, the FREE Dynamic DevOps Roadmap reached 1K starts 🤩
I'm personally happy about it because I've done mentorship for a long time, but after some time, things get repetitive, and I wanted to put that in a public repo.
I believe this roadmap differs from other roadmaps because it's Non-Linear and follows an Agile MVP style. Which focuses on understanding the problem instead of just learning a bunch of tools and then using an iterative style, where each iteration will cover most DevOps areas used in the real world.
I highly recommend taking a look at the FAQ page and appreciate any feedback from your side 🙌
https://redd.it/1g56luo
@r_devops
what should I use for logging/storing webhook payload in production, so I can replay and debug it later in case of failure?
I'm building SaaS with monthly subscription feature, using PayPal and its webhook. In development environment, I can use something like ngrok to capture webhook payload and replay it later.
I need something like this in production, but also to store it somewhere where I can check it and replay the webhook in production in case failure occur.
I'm thinking of using Redis to store the failed webhook payload for only the first few days it is stored, and then use Betterstack Logtail to log non-sensitive information, through pino transport.
Is there a better way to do this? I prefer something that can be all-in-one solution, but at the same time not too complex/powerful to use. I'm thinking of Grafana, but I wonder whether it is the best tool to use.
I don't have much experience in logging in production so I have no idea the best way and best practice to do this. To be honest, I was thinking to store everything in Logtail, including the webbook payload, but I think it's a security vulnerability.
https://redd.it/1g53ijx
@r_devops
Help on gitversion
Let me know if this is not the right forum to ask. If so I can delete the post.
I am trying to use gitversion tool for automating and controlling the semantic versioning.
This tool is giving me sleepless nights and I am looking at documentation and multiple blogs but still couldn't figure out how the rules work. Though I tried multiple channels i received no response and thought to check my luck here.
Can anyone give a example of these rules where I am looking for a few basics on how to control based on commit messages and how to control based on branch names.
https://redd.it/1g50lf4
@r_devops
Concerns with Windows Subsystem for Linux
The tools I work with often aren't cross platform and are intended for Linux only. Typically this comes from needing MPI support in python, etc. I've had a couple of colleagues warn against using and developing inside of Windows Subsystem for Linux, citing various concerns on "what it will do to my system".
After several months, I've had no issues at all. I'm able to use it just the same as my Ubuntu dual boots or ubuntu Raspberry Pi's, so I'm just wondering if the concerns are valid?
Anyone have any horror stories with WSL that would steer me away from using it?
https://redd.it/1g4ycdt
@r_devops
How do you make your staging applications private?
Specifically for simple applications with front hosted on S3, served by cloudfront, and back hosted on EKS/ECS/EC2. Still using an authentication.
I understand Cloudfront is not meant to be private, but how do you handle that questions? You can either have a staging infra different than prod (no cloudfront, and front running behind a container for example), but that doesn't make sense for CI/CD, repeatability and testing purposes in my opinion.
I have tried a solution with Tailscale AppConnector, but it's flaky and browsers cache DNS weirdly so traffic sometimes doesn't go through Tailscale app connector node.
How do you manage that? Do you keep your staging applications publicly available?
https://redd.it/1g4udag
@r_devops
Need advice on balancing project work and self study
I’ve worked in DevOps/SRE positions in different organisations for more than 12 years now. I need some suggestions on how you senior engineers here manage your daily schedule for self study and project work. Every day I want to plan some hours for self learning to upskill myself on different technologies but somehow it’s not getting happened mostly due to personal stuff. I feel a sense of guilt each day after my work that I’m not learning much outside of my work.
https://redd.it/1g4v1br
@r_devops
What does your development environment look like? What editor/IDE do you use and how do you keep your workflow as fast and easy as possible?
I once again upon Primeagen's brutally fast and systematized terminal only workflow where his workspace does exactly what he wants, when he wants it, using shortcuts and scripts he created himself. He doesn't need to touch his mouse for most of his work. I have to say that I'm pretty envious.
The thing is, devs mostly work on their own system, I more often than not work on another system or many systems in parallel. Next to the fact that I'm stuck using Windows at work. That causes me to use a random sandbox linux machine that I remote into using Cursor (VSCode) to use as my main workspace - to get at least some sanity back. But this does not "really" work, since many VSCode plugins just don't work remote and all my ZSH customizations also don't translate to any other server I have to connect to to get any work done.
I like it fast. Fast is good.
In the end, this job is, with the exception of writing actual code or automations, pure efficiency hell.
It is impossible to reach 'Flow' with all of these context and program switches.
How the hell do you guys survive this - or do you also suffer through it?
What does your dev environment look like, are you able to optimize everything?
Are you able to work without touching your mouse?
https://redd.it/1g4tfs7
@r_devops
Rate My Startup Idea: Cloud Management Platform Using Natural Language Queries
I'm developing a cloud management platform where users can manage AWS, Azure, GCP, etc., using natural language commands. The platform integrates an LLM (like GPT) to translate user queries (e.g., "Launch an EC2 instance") into API calls that perform actual operations on the cloud provider. This aims to simplify cloud management, especially for non-technical users. What do you think of the idea, and how could I improve it?
https://redd.it/1g4rwwz
@r_devops
Folder structure, Deployment best practices with bicep across multiple subscriptions with azure devops. Help
Hi Guys,
I'm kinda new with Bicep and I'm working on the folder structure, for Infra deployment with Azure DevOPS. This is my current folder structure. It seems I'm missing something that I cant point out . I tried googling it but I need some feedback. Anyone with experience? Thank you.
-infra
--->project1
-------main.bicep
-------params.json
--->project2
-------main.bicep
-------params.json
--->project3 (same subscription with project2)
-------main.bicep
-------params.json
-modules
-pipelines
-scripts
https://redd.it/1g5r4a9
@r_devops
How do you handle complex data wrangling in Google Cloud or Azure Data Factory?
Hey all,
I’m working with Google Cloud Dataflow and Azure Data Factory for complex data wrangling. Any tips for optimizing pipelines, especially with large datasets or intricate transformations? What strategies or tools have helped you improve performance or streamline processes?
Thanks!
https://redd.it/1g5o6j8
@r_devops
Pivoting towards being Network Focused DevOps Engineer
For context, I have a bachelor's degree in Computer Networks and have had a network engineer role for about 18 months before being let go, and I am currently searching for roles.
Would it be wise to pivot towards Network-Focused DevOps roles given the lack of computer networking jobs?
I know that lulls in opportunities can be temporary, I'm just considering my options.
I understand there's been a shift in the fact that there's a lot of network automation as well as a heavier focus on cybersecurity.
https://redd.it/1g5khgg
@r_devops
Good devops oriented networking training for my team?
Hi guys,
I’m the lead of a team, and as it usually goes, it seems like most juniors and even more senior people, who are very qualified at cloud and whatnot, lack basic networking fundamentals.
They don’t truly understand what NAT is, what the difference between routing and bridging is, what peering is, route propagation, etc.
I’m looking for a certain course / training that I can give all of them so that at least everyone in the team has some basic foundational knowledge about networking. Eg if I ask them to set up a redundant wireguard tunnel for site-to-site peering between two different clouds, they know at least understand what they’re being asked to do.
There’s so much available online, but the Cisco etc types of trainings are way too much.
Does anyone in this community has any recommendations on a good ~ 3 day course that goes over the fundamentals?
https://redd.it/1g5jw9k
@r_devops
Transitioning from Software Developer to DevOps, looking for advice on finding my first role
I'm a software developer with 3.5 years of experience with creating React/.NET/Java/Kotlin applications for enterprise companies. I started to realize I wanted to try something different, so I started looking for another role. Was looking around for dev roles with a different tech stack and had no luck.
I talked with a few people from tech networking events and over time I gained an interest in working with AWS/DevOps. I went for the developer associate certification and passed after a couple months of studying. I did a lot of hands on labs from Adrian Cantrill's courses and have a solid grasp of it. I also learned some networking essentials while doing that, but I know I still need to review/refresh my memory on some concepts.
I started applying again but am not finding many DevOps/related roles at the moment. I'm seeing most of them are contractor roles, and the full time roles have pretty steep requirements (i.e. senior with 8+ years experience).
Outside of applying, I've been making time to learn Jenkins, Terraform, Docker, etc. to help boost my resume a little bit. I understand the job market right now is a bit rough right now, so that's likely a big factor. I do have a solid Linux foundation from my previous roles, so I know the CLI and how things work fairly well. I do list my side projects on my resume mentioning how I use the technologies.
Unfortunately my current role doesn't have any devops roles internally or any opportunity to do so, and I'm looking to leave due to office politics.
A couple questions I have:
* Is contractor the way to go if I'm trying to get into a Devops role? I do see a couple of "Junior contractor" devops roles, but I'd rather go for a full-time vs a contractor. I'm a bit hesitant being a contractor since I don't know how friendly/what the expectations would be for someone with no Devops work experience.
* Is there any other "titles" companies give a devops role? I've searched for Platform Engineer, Cloud Engineer, SRE, Software engineer (infrastructure), etc.
* Any general advice? Is there something I might be missing/neglecting? I know I'm missing some of the skills needed, but I'm trying my best to learn them and create projects out of them. Happy to answer any questions if needed.
https://redd.it/1g5el6q
@r_devops
📋📣 Ansible Issues and Challenges
Hey everyone!
We are a joint team from Carnegie Mellon University and Instituto Superior Técnico and we are doing a study to explore the challenges and issues DevOps engineers face while using Ansible. In the future, we want to create tools that help developers with these challenges.
If you work with Ansible and you are interested in ranting about the pains this wonderful technology has caused you, click here to participate in our study:
https://cmu.ca1.qualtrics.com/jfe/form/SV_5argDxTDTGq8jfo
If you agree to an interview, you'll be entered into a $100 gift card raffle! 💸
Thanks!
https://redd.it/1g5cowz
@r_devops
Any protips for getting an interview? Recruiters?
Background: CS degree, 7yoe, another 7 in automated QA. Mostly Azure, have az-400 cert. Resume gets past ATS for jobs ive applied for and i'm at least 80% qualified for. I do all the automated thing from IaC to CaC to pipelines to cost control, policy/governance and custom programming. American, EST and yes applying for remote roles that are fresh (1-2 weeks max, usually <3 days).
Is stuff just SUPER saturated w applicants these days? I'm applying directly on companies websites mostly.
Any recruiters want a good candidate? Any other protips?
TIA!
https://redd.it/1g58mz1
@r_devops
DevOps coding interview in 1 day
Hey guys, I have a coding interview for a DevOps position. They said I can code in my language of choice. For it’s Python. However, I am a little rusty, what should I focus on giving it’s only a day left ?
https://redd.it/1g5643s
@r_devops
40+ years old, just landed a Devops job about 6 months ago: I don't like it. Advice?
Hey all. I don't know if this is the right place to put this but here it is. I'm 40+ yo and stumbled into a DevOps job with a pretty reputable company with great benefits. For many years I was just a Systems Admin/engineer just kind of wadding along. Covid hit and throughout that time I was laid off twice. I then got a job doing something really left of what my career was for about 2 years. Was in peril of getting laid off and got a job within the company even further left of my career and it took a toll on my mental and physical health. I resigned and I stumbled across my current gig and the people are great, the company is good, and the benefits are up there. My issue is that it is Devops. I never really did anything in the DevOps vein. I barely had to script/code anything throughout my career. So, I am happy to be here and happy to learn but I feel like this isn't my thing. It is one thing to learn ONE technology at a time (maybe even two) but I am being asked to switch focus every couple of weeks it seems and I can't seem to get good footing on anything. One week it is grafana issues, another it will be getting into Terraform, and then another week it will be Kubernetes. I put my age in there because at my age and with kids, I find it hard to have the energy at the end of the day to learn on my own time. But even when I am learning on my own time, the focus changes often. Is this normal in Devops? should I be concerned? Funny thing is my wife is in Devops and she, for the most part, loves it. I think my main issue is that I didn't start from square one. I dead smack into some projects that already grew legs and has some foundation but I'm having a difficult time maneuvering through the environment. Especially with people training me but not at ground level. They don't know what I dont know but at the same time I feel they don't want to assume that I know nothing...which wouldn't be that far off haha. What are your takes on this? I am aware of "impostor syndrome" but I think I may be a bit different. Or I assume so and this is normal for budding DevOps Engineers. I originally wanted to just be a cloud engineer. Had received my AWS certified Solutions architect associate and wanted to build on that. Do you think I'd potentially run into the same issue that I am having now?
P.S. Give it to me raw. I know some of you are going to be brash about it anyways. Reddit has no shortage of folks with no empathy haha.
https://redd.it/1g52c21
@r_devops
Need Career Advice: Are AWS SAA, CKA, and Terraform Associate Enough for Job Hunting?
Hi everyone,
I recently graduated as a new grad from the University of Waterloo with an engineering degree and passed the AWS Certified Solutions Architect – Associate (SAA), Certified Kubernetes Administrator (CKA), and Terraform Associate exams. I’m currently exploring job opportunities in Canada, but I’m wondering if these certifications are enough to start applying directly for cloud and DevOps roles.
If not, I’m planning to pursue more certifications and would appreciate your thoughts on the order of priority:
1. Microsoft AZ-104 (Azure Administrator Associate)
2. AWS DevOps Engineer – Professional (DOP)
3. Certified Kubernetes Application Developer (CKAD)
Do you think this plan makes sense? Or would you recommend a different approach? Any feedback or insights would be greatly appreciated!
Thanks in advance!
https://redd.it/1g4zntp
@r_devops
What would be a "magical" tool to resolve your problems ?
I’m curious—what’s a tool that you wish existed to make your job easier? Something that could solve those recurring problems that current tools don’t quite handle. Whether it's better automation, monitoring, CI/CD pipelines, or integrations, I’d love to hear what you think is missing!
What tool would make your day-to-day smoother?
https://redd.it/1g4xlt5
@r_devops
Getting started in deployment
I am a Full stack software engineer who has decent experience with JS frameworks. Now I am planning to deploy a MERN app on digital Ocean. Can anyone please tell me how I can learn deployment on the Digital Ocean? I have a basic idea about Ubuntu so there I can clone repo and run it on localhost but have no idea how to connect with domain and other things like PM2 can anyone please guide me on how I can learn all these things?
https://redd.it/1g4vznc
@r_devops
Best practices of using terraform in local machine or a vm
I am having some questions regarding this , since most of the people I know are using tf locally but I created an instance to use tf , since its on the web i can access it from wherever I want , how do u use urs ??
https://redd.it/1g4rcxx
@r_devops
Seniors who took a chance on a junior hire, how did it go?
Did it pay off? Where are they now? Just curious of other people's stories.
https://redd.it/1g4sqg3
@r_devops
Need resources for DevOps beginner guide
Hey there,
Given my background in deployment and automation, I want to explore devops career path.
Started with few udemy courses and hands on with docker. Exploring further on kubernetes , ansible, aws cloud etc
Let me know good resources to start with for the same
Hoping a positive response.
https://redd.it/1g4pq1c
@r_devops