86
Reddit DevOps. #devops Thanks @reddit2telegram and @r_channels
Job Market Woes - Senior-level
I’m curious if anyone else is currently experiencing a frustrating job market (United States, east coast but looking for remote work). I do realize that things are not the best right now with the 2023 layoffs, RTO at many companies etc. That said, I have what I would consider pretty solid credentials and not even one company has gotten back to me after over two months of applications.
Background: I have been working for AWS for nearly 10 years, the past 5 or so working my way up the ranks in a solutions builder team. I do tons of CI/CD, IaC, serverless, containerization, you name it. My most recent work has gotten me deep in backend OOP as well, so I feel that only helps my case. I have all of these skills clearly laid out in my resume and what I believe to be a pretty strong profile paragraph. I have several industry certifications to back up those skills.
Is the job market really that difficult right now? I’m concerned something with my background or resume just isn’t clicking, but I don’t know if hiring a resume writer will be a worthwhile investment. I have been applying to several Sr. DevOps/SRE/Cloud Infra type positions where my qualifications match nearly every statement on the job description. I’m not overly concerned with working FAANG jobs anymore and have been applying at both SMBs and larger corporations.
Thanks for listening, and any tips would be much appreciated!
https://redd.it/18kgxne
@r_devops
Newcomer advice
I volunteered to become a member of the platform team at my org. I’ve never done DevOps before - previously I was a front end developer. I’d never really even used a container.
Holy shit. The amount of stuff to know is mind-numbing, and each day I swing back and forth between giving myself pep talks, and telling myself I should give up and go back to front end.
Any advice? Any books to read? Websites to visit?
https://redd.it/18i0tiq
@r_devops
How do I convince my org to adopt Opentelemetry
I work for a fairly large org and we manage and provide many kubernetes clusters as a service(s). We implement logging and metrics in a very straightforward way using the ELK stack and prometheus+grafana. We dont plan to move away from these anytime soon.
We do have plans to implement tracing via Jaeger and long term metrics storage via Thanos.
What benefit would we get by implementing Opentelemetry as a collector. Would it just be another layer that we have to understand and manage?
I would have to pitch it and sell it to higher management on the benefits of adopting Otel.
https://redd.it/18hi8dp
@r_devops
How to add basic monitoring to resources only reachable through a VPN?
Hi folks, I've been trying to set up New Relic basic ping monitoring for a couple of web apps, both front-end and back-end. Production front-end can obviously be reached without a VPN, but sections of its API can't. Everything in Development is behind a VPN as well.
My question is, is there a way I can create a monitoring setup just to know if they're online (like with updown.io) through this VPN? I've been thinking about creating a bash script in the VPN instance that pings these sites but it seems like a rudimentary solution that would take time to scale.
Any help is greatly appreciated.
https://redd.it/187jhsx
@r_devops
hardest thing to find in a DevOps hire
Having been through multiple recent bad new hires in our company, I got to thinking about what is actually really difficult to find in the hiring field. It's not finding experience in cloud, or in a specific tool, or even a specific language. It's not someone who has experience in kubernetes (although an actual SME in kubernetes seems to be actually rare), or terraform.
It's really just...someone who is personally competent enough to put all of these things together in a way that actually provides value. I think everyone takes a different amount of time to scale up and get comfortable in a new environment, especially one like mine where there is a lot's of legacy stuff not well documented. However, it just seems like people have these bits and pieces of information floating around that they can access with no real substantive connectedness that results in meaningful resolutions.
I am talking about someone who is presented with something they've never seen or aren't familiar with, and can fit that into their knowledge bubbles and give a good estimation of what should happen regardless of the specifics. I can't understand how senior DevOps engineers who supposedly have 7+ years of experience still need guidance on how to do simple requests or can't actually take ownership of a process from start to finish.
I am also not talking about just people who want to learn or who are quick learners. There are people on our team who are curious and want to learn as well, but still need lots of guidance.
​
I am guessing this is the case in any field, you just want someone who is competent and has a good head on their shoulders. I didn't mean for this to turn into a rant, but ...rant over!
Edit:
Lots of people seem to think I am saying that every DevOps engineer should be an expert in everything. I'm not! That wasn't the purpose of this thread. You can be a very competent engineer and only have 1 or 2 areas you are an expert in. It's all about how you approach things, how you communcate, and your ability to grok new information.
https://redd.it/185vros
@r_devops
Terramate vs Terragrunt
I am curious if anyone has experience with both Terramate and Terragrunt or migrated from one to the other.
I am currently using Terragrunt but am interested in potentially making the switch to Terramate. Would love to hear feedback from the community on Terramate.
https://redd.it/17y91ib
@r_devops
Running production workloads on K3s
Hi, Just wondering if anybody has tried this. To me it looks really useful for my own dev environment. I understand that it can be used for production workloads but I am not sure how that works in practice. If you are running it in the cloud, what sort of platform would you host it on - a VM, a container service, inside K8S itself. Is there a lot of overhead in managing networking, availability etc?
https://redd.it/17s0zbd
@r_devops
Should I learn DSA(Data Structures And Algorithms) If I'm learning Devops?
As the title suggests, Should I learn Dsa? I heard it helps in interview so as I'm starting with devops should I also consider DSA alongside with it? Is it ok If I don't learn DSA for Devops?
https://redd.it/17o4prz
@r_devops
GitHub announces future Arm64 support for GitHub Actions
https://github.blog/changelog/2023-10-30-accelerate-your-ci-cd-with-arm-based-hosted-runners-in-github-actions/
I'm really looking forward to this - it's really going to help everyone who wants to run their apps on lower-cost, higher-performance Arm64 instances in the cloud.
https://redd.it/17lgfrz
@r_devops
Systems that support WAY more revenue than they should
I know you've got war stories. Tell me about systems made out of shoestring and duct tape that yet somehow supported a ton of revenue. I'll start:
One of my first contracting gigs many years ago was with a marketing analytics company. They'd ingest data from your ad vendor, and then use that impression/conversion data to evaluate how effective different ads are and then provide consulting services with that data.
Their "infrastructure" was a single 4xlarge AWS instance. It contained:
Their only MySQL database
The entire ETL pipeline
The client SFTP dropbox
An nginx server with a handful of ingestion API apps
Again, this is a single server running literally their entire tech stack
And of course that self-hosted MySQL had no backups. And of course the deployments were done by FTPing into the server.
A 4xl is a fairly large server. But for $1000/mo those guys pulled in $8m the first year I was with them.
https://redd.it/17hdh3i
@r_devops
Brute force to make things work
Hi,
Is it common to spend 10+hours trying to force an infra or a config to work? I often do this, and lose sleep while doing it. I would just try different configs, each iteration might take 30m+, but I'll just keep training, brain dead, until it works.
For example just now, I've been trying to make this SageMaker deployment work but it just won't. No up to date documentation for what I'm trying to do. I just don't know what to do. I can't really debug it because it's a managed service. I tried AWS support but they are too slow and not helpful most of the time.
What's the right thing to do here? I don't really have anyone to ask I'm on my own.
https://redd.it/17fyjit
@r_devops
Are all managed Kubernetes clusters created equally?
I've compared using EKS to AKS as the platform to host a development Kubernetes cluster, including CI/CD using Tekton and ArgoCD. The goal was to make the developer experience identical for both. In this blog post, I've used a real-world use case as a basis. I've tried to capture the experience, the similarities and the differences. How are you deploying application landscapes to different versions of Kubernetes on distinct cloud providers? How did you allow the usage of vendor-specific services without having to create different deployment configurations? Or what was the reason for using vendor-specific deployment configurations?
https://blog.ordina-jworks.io/cloud/2023/10/20/k8s-comparison-azure-aws.html
https://redd.it/17cvyxm
@r_devops
What is expected from a DevOps Engineer in 2023 (almost 2024)?
Hello to everyone reading this! I wanted to make a check up on my progress this year. I've been actively searching for a new position in DevOps throughout the year, and I've been dedicated to honing my skills to stay competitive in the market. Despite going through numerous application processes, I haven't have not landing into a new position yet. It leaves me wondering how much of it might be my own doing.
All of us in this subreddit have a solid understanding of what DevOps is, and for those of us with more than 3 years of experience (YOE), it should be crystal clear. However, I'm not entirely sure if the job market is still a harsh as it was in the first half of the year.
My tech stack includes a range of automation and deployment tools to cover the entire DevOps process. Here's a breakdown:
* Ansible, Terraform
* Jenkins, Gitlab Actions, AWS Pipeline
* JUnit, Selenium
* Docker, Kubernetes
* MySQL, PostgreSQL
* Experience on both Linux and Windows Server
* General AWS knowledge to deploy, maintain and supervice a pipeline on their services
* Also some experience on some ETL tools like Pentaho, Kaftka, etc.
* Of course scripting with Python, Bash, CMD, etc. and experience with POO on Java and little C#
With this stack I don't know what am I missing, maybe more experience on Cloud but I've practcing also (AWS mainly) and worked on mid-large enterprises so also I have experience working on engineering and not-engineering teams, so my questions are:
Should I keep working on those skills to be relevant to the market or should I focus on other stack?
Is it the market as we see on many other positions or I am being incompetent at the interviews?
I am located in Mexico so I get offers from a lot of american companies and mexican ones aswell.
Any feedback, comment or recomendation is very very welcome!
​
https://redd.it/1750yj6
@r_devops
Most effective job boards
I was let go as of this morning but I've had applications on a slow drip prior to this with responses around 10% of the time. Now that applications are going to be my full time job until another comes up, and I want to maximize my ROI. I've mainly stuck to LinkedIn and applying directly to companies I've interacted with before or get directed to by my network, but this isn't getting the amount of traction as I hoped. I'm confident in my resume as it's gotten me through to previous interviews/offers with AWS and MS so I'm pretty sure it's not a weird resume quirk.
Are there other job boards out there that have been more receptive or produced better results for the amount of applications you send?
I'm in the DMV area with about 5 YOE working AWS-based applications in private and government sectors. I think my biggest drawback is that I haven't had production experience running k8s for any of my previous roles.
https://redd.it/171n6np
@r_devops
Do you ever stop getting thrown in the deep end?
At my current job when I am assigned a project it feels like I am totally thrown in the deep end without any assistance. Often times I have found that it is flat out impossible for me to finish my tasks because I was missing some key piece of information that was on some esoteric confluence page that I had no idea existed. The on-boarding at this company has been non-existent and the interview in no way represented what the job was like, interview questions were about k8s deployments and replicasets and I can currently tasked with swapping out kubernetes internals which I've never had to do since I've only worked with AKS & EKS before.
I am used to having some uncertainty as I pick up tools and learn my way around the company environment but at this job I feel totally lost, unsupported, and out of my depth. Is this the norm or have I just been lucky at my previous positions?
https://redd.it/16zyhwf
@r_devops
TIFU and took down production
So I'm doing Kubernetes cluster upgrades at the moment. Updated dev earlier this week, all ran smooth except for some PDBs blocking node upgrades (we have, for reasons too tedious to explain here, some workloads that don't play nice with PDBs). This morning came time to do prod. So I get up at the crack of dawn, grab coffee, manually delete the PDBs I know will be a problem, and fire up the pipeline that runs the TF code that forces a version upgrade on the prod Kubernetes cluster.
Get an error relating to an ingress config. Remember that this issue was encountered in dev yesterday and fixed by a code change. Pull the infra code, merge from dev branch to main branch, wait for Flux to reconcile the infra, then re-run Terraform. Cluster upgrades. Cool.
Then the alerts start to fly. All of prod is down.
Turns out there was significant drift between dev and prod infra, to the point where dev branch IAC flat-out breaks prod when merged. Also turns out there was a half-completed refactor in the IAC code that, for reasons known only to Satan and the original developer, works on dev but breaks prod. Also turns out that the refactor nuked Kafka on prod only. Also turns out there are various other issues with DB pooling, Kafka topic replication, and time-outs that only became apparent when this shit hit the fan.
Three hours of fault resolution later, everything's back up and the postmortem's done. Main takeaway is that whoever's refactoring IAC code now has to block merges to prod branch until the refactor is done and tested. Outage blew a big chunk of our error budget for this cycle. Not sure whether this will have consequences for me or not; I've already done the apology tour and been met with understanding comments by the rest of the team, who know what sort of a overly complex tech debt minefield we're dealing with, but I'm sure the client is not happy at all. Will no doubt hear more about this next week.
Now I'm drinking, wondering whether it's too late in my life to change careers and become a competitively-priced prostitute, and contemplating not touching anything more technically complex than a hammer in future.
https://redd.it/18j1gyc
@r_devops
Thank you for Resume Feedback
Hello all,
Just wanted to quickly share a huge thanks to everyone that provided feedback to my resume. After a handful of iterations I have gotten 3 interviews in the last 4 days.
Just to recap, I had gotten about a total of 6 interviews since early October (I was laid off September but did not start actively looking until early October). So 3 in the last 4 days is an amazing number.
For anyone that is looking for a job, regardless of reason, I would suggest asking for feedback on your Resume and taking some advice from others here. Also, BE PATIENT. It is the end of the year, even if the job market wasn't this hectic, it would still be SLOW.
A lot of managers and "decision makers" take a lot of time off during this time. Honestly, I am expecting to land a role sometime in February (hopefully sooner if I do well in these interviews).
Another tip is to take ANY interview, even if you don't want the job, it is good practice. It lets you take a great look at yourself and see what you need to work on.
Just one thing more, during this downtime, try to not be so hard on yourself. While you are doing your job search LEARN something, anything. Things that you would like to learn just to learn, whether it will be helpful in your job or not.
There will be a time where all you do is work and you will wish you had the time to do other things.
Hopefully, my next update will be letting you guys know I landed a role.
Happy holidays everyone.
Edit: I am adding my resume here: https://imgur.com/ycFsa3p
https://redd.it/18hpj2b
@r_devops
Company considering moving on prem, and off cloud
Hi! How would you guys handle your company considering this a move from cloud services like AWS? I’m an IC with 15 yoe, so I’m smart enough not to get into the politics of why we would move (it always just moves cloud costs to personnel costs).
If it goes through I’ll likely find another job once/if progress actually starts (you know how that goes). I’m just not interested in a skill set of such limited use and I’m not sure I trust any internal IT organization to understand the ins and outs of hosting Kubernetes clusters, Postgres replication, software defined networking, etc when they have actual IT work to deal with. It’s been 13 years since I worked with a non AWS or
GCP environment and even that was hosted by Rackspace. I thought this crap was over!
Basically I don’t even want to run the risk I get stuck with some sysadmin work bc we want to move a few bucks to another cost center.
Would a move on prem make any of you leave, or am I being too over dramatic?
https://redd.it/18awgos
@r_devops
Being made redundant, but the company wants me to stay on for another 9 months.
I've been working for the same company for quite a while, across a few different roles. I started as a jr. software engineer before moving into an assistant product management role, and most recently I've moved into a jr. DevOps engineer role. All of this role hopping seems to have bitten me in the ass, as I've never stayed in one role long enough to move past a 'junior' position.
It was recently announced that all of our roles are being outsourced to Malaysia. They've asked us all to stay on for another 6 months during the transition. It was a bummer, but in exchange for staying for the 6 month period, I'll be getting a severance package worth nearly a year's salary.
That's all well and good, but it seems they've suddenly realised what a disaster they're going to have on their hands, so they've come back to me with a new offer. After the 6 month period is up, they want me to stay on for another 9 - 10 months to lead the tech support team.
It's a job I could do with my eyes closed, I'd still be getting paid my current salary and my pay-out at the end of the period would remain the same. I'm struggling to figure out whether I should accept it or not.
I want to remain in a DevOps position, but currently my qualifications are a little lacklustre, which is making me a bit nervous about job hunting. I have a year and a half worth of DevOps experience, an AWS cloud practitioner cert, and that's about it.
I've been studying for the AWS solutions architect associate exam, and I'm somewhat confident that I'd be able to get it before the end of the initial 6 month period, but there's also the chance that I won't.
Accepting the position would give me some breathing room, but I'm not sure if having my last 9 - 10 months worth of experience being in a tech support role would hurt my chances when applying for a DevOps role.
What would you do in my situation?
https://redd.it/186dsqg
@r_devops
Do you use “non” standard tools in your daily basis?
The idea is to share those tools that you use and really know that are non standard, but for you are really valuable for some reason.
GUI ones or cli.
In a container or installed in the OS.
SaaS or self hosted.
All are welcome.
In my case for example, I use everyday Memos just for own documentation, snippets, comments, etc. Is a well-known tool, but isn’t a standard.
https://redd.it/181is6d
@r_devops
What makes a good Senior Engineer?
I’m in contention to being promoted to a senior in the next couple of months, I have great communication skills, I meet the deadlines almost everytime, help other people, try my best to follow best practices and avoid anti-patterns. I always get my shit done, but I work on too many technologies that I seriously don’t know what I’m good at and this where the imposter syndrome kicks in.
All of this is great, but I don’t think I have the technical skills yet to become a senior engineer. One of my colleagues is at the same level but they are super smart, writes crazy nice code and does crazy stuff, but they often drift from their task and have trouble communication their progress and deliverables.
I really think that my colleague is better than me, and I think they deserve being promoted before or with me, but me being their senior doesn’t make sense to me. Should I mention this to my manager?
https://redd.it/17sw0ao
@r_devops
Calculating a company's required DevOps Capacity
A nice blog with formulas for calculating how many DevOps Engineers a company should hire: "Calculating your company's required DevOps Capacity"
https://redd.it/17p403p
@r_devops
I'm expected to do a CodeSignal assessment for a devops engineer interview???
Anyone else run into this before? From the practice questions, it seems like standard "Leetcode" style questions, not devops specific ones. I have a decent amount of coding experience, but it's all like working with apis, shipping logs, authentication, querying elasticsearch/mongo, etc.
Seems kind of absurd. The job description doesn't even place a heavy emphasis on coding. The pay is great so I'm debating griding out some practice questions to see if I can at least solve 1 or 2, but, ffs, I'm not a software dev.
https://redd.it/17lij2s
@r_devops
Is it worth learning service mesh?
It seems like an incredibly cool, and potentially powerful, technology.. That said, I feel a bit like learning it would be solving a problem I may never actually have.
I've worked with customers who use it. But I'm not really aware of.. Why.
Maybe it's me. Perhaps I should quit my job and go work in a helpdesk. McDonalds is hiring. I dunno. I don't think McDonalds cooks need to learn service mesh. But then, I don't understand anything about service mesh, so maybe they do and I just don't see it.
What is service mesh?
​
.. Are you a service mesh?
https://redd.it/17l3jw4
@r_devops
Personal project ideas
Hi,
I am currently trying to break into the DevOps landscape, but I need some personal projects to spice up my resume a little.
I am comfortable in Azure (+DevOps), GitHub (+Actions), Terraform, Linux, basic Docker, and basic PowerShell and Bash scripting.
I am currently learning Python, and I want to expand my knowledge further by learning Kubernetes, Docker, Ansible and a monitoring tool like Grafana aswell.
So what I need, is a project that utilizes these tools together. Is this possible? If so, please let me know of some project ideas in your brilliant minds. If needed, I have a simple homelab that I can self-host anything.
https://redd.it/17g8rgu
@r_devops
How necessary is AWS certificates
Hi everyone.
I am a senior CS student and I have been working as a fullstack developer at a local company for the last 1 year. I have been working with React, Node, Java and AWS. I don't really have a defined work and do frontend, backend, devops and even web scraping when necessay.
Mainly, how I deal with AWS is, when I need something I just look it up and implement it. I have hands on experience but I think I lack the theoretical background. I am thinking of completing an AWS course on Udemy, and I was wondering if the certificates are useful anyway. Should be going for AWS Cloud Architect certificate? Is it valued in the industry?
Thanks im advance.
https://redd.it/17dmblv
@r_devops
Tools for Platform Teams @ Pulumi
Pulumi launched a set of features built for platform teams today. I think these tools can be used for agencies too! Having ways to standardize and document things is great!
I’m curious what people think!
https://redd.it/175mk2o
@r_devops
From devops to engineering manager ?
Hi all,
I like my sre position, but i am realizing that as time go by and as it is very exhausting, i do not see myself doing that forever.
Hence the title, going up the managerial ladder, first thought was engineering manager,as it regroups soft and hard skills that need a technical and broad understanding of the app distrubution as devops often do.
What would be the pros and cons ? Did any of you guys switch from devops to em or to anything else for the same reasons ?
Thanks a bunch :)
https://redd.it/1720whz
@r_devops
RANT/VENT - Frustrated about "DevOps" as a job title
Quick background : I started in Help desk -> Network Admin -> Sys Admin -> DevOps
I have been a "DevOps Engineer" now for 4 years. What that means is in my DevOps role I've worked with everything from Zabbix to K8s clusters, Rancher/Argo/Docker, Prometheus, Cisco Routers/Switches, Azure, AWS, GCP, CI/CD pipelines, Serverless functions, AI chatbot setups with Cognitive Services and D365/Powerplatform, Ansible, Terraform, etc, etc, etc.
The only thing I haven't done is coding.
I write Terraform Repos, ARM templates, CloudFormation modules, I write Python, Shell, Bash, and C scripts for network checking tools with Netcat, SNMP gets, or to provision things, compare PAT tokens and keys, or write a C# service layer script between a Javascript web app and an Azure CosmosDB.
It's a bit of coding, but I would say like 10% of my job is coding/scripting, and I am actually NEVER tasked to do any of it, I voluntarily do it on down time to practice.
I've been casually browsing DevOps jobs, basically recruiters will call me asking me if I'm interested, and I tell them "I am interested in the job if I can code more and be involved more with the development/software engineer team so I can improve my DevOps skillset".
It's been failing every time, recruiters have consistently been saying "this devops job is not coding related", and they list off job duties of just a sys admin..
and then they say things like "we arent here to help you do a career change" or another one said "devops isnt coding, and i dont understand why you want to leave devops to be a developer"
​
I'm very confused here, is DevOps NOT a Developer and an Operations Analyst in one ? Does nobody know what DevOps mean? Or do I not know what DevOps means? Is it SO fucking alien of me to try learning how to code as a devops engineer?
​
TL;DR: "DevOps" is a worn out job title, recruiters and hiring managers think its the new cool word for Sys Admin, and now I have to go for "Platform engineer" jobs to get a real DevOps job.
https://redd.it/170n1zr
@r_devops
What the fuck is Devops
Got switch from Software Engineer team to DevSecOps. Backlog is all over the place from pipeline to data analyst to Security. Do people in DevOps do everything? How do I even get start ?
https://redd.it/16oowkr
@r_devops