86
Reddit DevOps. #devops Thanks @reddit2telegram and @r_channels
The Journey of Code to Production
Wrote a blog post that should be interesting for junior developers curious about Ops side of things. I reflect on my journey developing Java code and learning to deploy it through Containers, Kubernetes and ArgoCD. The aim is to gain high level understanding of why these tools exist and why you should know them without going into details. Hope you find this encouraging! Feedback welcome.
https://itnext.io/the-journey-of-code-to-production-588990234b78?sk=c72c63f2a53c9c27c10f875c7330cdc2 (friend link)
https://redd.it/1camkgk
@r_devops
Accessing control plane of private GKE cluster from another private cluster
[CONTEXT\] Up until this point, I've had my self-hosted runners running in a separate namespace in the same cluster as the apps. I use the self-hosted GitHub runners to do all sorts of things: deploy GCP resources, deploy helm charts into the cluster, deploy database resources.
I'm adding the staging environment and I'd like to separate the runners into a different GCP project, different cluster.
When the runners were deployed within the cluster, it was obviously no issue to install helm charts or create kubernetes objects from the runner. Now, the runners would be running in a different private cluster, within a different VPC.
I was initially thinking to peer the VPCs that the clusters are running on and try to configure terraform (running in the runner cluster) to connect to the dev cluster (where the resources should be deployed). However, Google runs the control planes of GKE clusters on their own networks, and they peer those networks to your VPC, where the workloads run. That coupled with the fact that they don't support transitive peering means that even if I peer the VPC where the workloads are running, it doesn't mean that the runner cluster will have access to the control plane of the dev cluster.
Just as a little aside, I'm not sure if what I'm trying to do is a good idea or not, so please feel free to tear it apart if you know a less contrived way to do this.
https://redd.it/1cagi3w
@r_devops
Need
Hi everyone, I have 2 yoe , currently in support. I want to move to Devops. Please guide me. I know the roadmap and basic docker, K8s. I am planning to take coaching either in Bangalore or hyd. Please suggest me... I tried learning from YouTube but I don't feel that confident.
https://redd.it/1cacfw9
@r_devops
DevOps Roadmap & Learning Path 2024?
Seeking insights on DevOps learning paths & roadmaps for 2024. Let's exchange ideas and explore the best strategies together!
https://redd.it/1ca9s7k
@r_devops
Grant Kubernetes Pods Access to AWS Services Using OpenID Connect
What do you do when you want to grant Kubernetes Pods access to the AWS account?
Do you create an IAM User and pass the credentials as a secret to the Pod?
Well, I'm here to tell you that there are better ways.
I haven't seen many people talking about this, so I created my own.
Beware that the AWS EKS already has a detailed guide on this topic. However, you will not find tutorials on how to grant bear-metal Kubernetes Pods access to the AWS services.
This post is here to address that.
Enjoy and share the love with your network. 🐧 🦀
https://developer-friendly.blog/2024/04/22/grant-kubernetes-pods-access-to-aws-services-using-openid-connect/
​
https://redd.it/1ca5lau
@r_devops
I'm trying to configure a bitbucket runner to run continuously on macOS
Hi guys, I need a little help.
I created the runner and added it to macOS and when I run the start command it works. But I would like to make it run continuously. And I saw that I have to create a config file in LaunchAgents to run continuously. I created it but it doesn't run. I ran it with the command launchctl load name_file.
With this two files I tried: file1, file2 .
I looked for tutorials on this but there aren't many. Can anyone help me with this?
https://redd.it/1ca4itm
@r_devops
Looking for best cloud solutions
I have been researching cloud computing and cloud services for a startup company related in media and ad servers plus web hosting services. The required clusters would includes the following:
- Two cloud Computing servers running on Linux (to start). These servers would be used for WHM/Cpanel.
- DNS server
- Databases running MySql
- Firewall
- Loud Balancing
- Linked with CDNs.
I have been currently looking at GCS and AWS.
https://redd.it/1ca16cw
@r_devops
Coming over from Azure, which AMI for containerised apps?
Hey,
Coming over from Azure to AWS and see that you are required to choose an AMI. Typically I run everything through containers, so which AMI do I pick?
There seems to be 'Deep Learning' ones as well? This is just the base image of the VM which my container won't even interact with right? Since the docker file's 'FROM python:3.8-slim' will take over?
Thanks
https://redd.it/1c9vref
@r_devops
Need help on project ideas to go more in depth with Terraform and Ansible
Hey all! So for the past few months I'd say I've gone pretty in depth on Terraform. I've learned the best practices, how to write good modules, optional resources, seperating environments, etc. I learned this by actually implementing projects that I enjoyed working on and found useful. (My most recent: https://github.com/foulscar/websitecv). The problem is, I want to learn more about Ansible as well but I just can't find or come up with any ideas that feel "relevant". I loved learning Terraform because it felt like I was actually building something. When I watch videos or tinker with Ansible, it just feels like patch-work. Granted, I know that's one of it's biggest use cases but I also know there is so much more you could do with it. I welcome criticism and if anyone has any ideas or even a different way to think about this, it would help a lot. Thank you.
[Edit\]
For reference, I noticed the idea of orchistrating multiple minecraft servers seems pretty fun to me, I'm hesitant on that though, because i don't want to spend more than warrented time learning plugins or java (I like golang and Python) because my main focus right now is on learning relevant skills. At the same time though, I already have the basics of docker down so doing that and maybe learning Kubernetes would probably be extremely fun.
https://redd.it/1c9ofhe
@r_devops
Continuous learning and sanity
There's always something new to learn and master in DevOps. To stay employable one needs to learn many of those tools/systems.
It results in many situations when one takes (in his/hers mind) too long to finish new tasks, needs to ask for assistance or just reply "dunno".
What to do to stay sane, not beat oneself up and look plain stupid to colleagues or management?
https://redd.it/1c9o56v
@r_devops
Suggestions about this GitLab Pipeline
I'm not a DevOps at all, but I have asked a friend to make me a Gitlab pipeline that compares the Lighthouse Accessibility Score of **"Main" Branch** *with* **"Dev" Branch when trying to merge**. If the score is lower, it will **not** allow the *merge*, otherwise it will allow it.
*It works, but I have several doubts:*
* What is the first line "image" used for?
* What is the last line for? The thing with the "Curl"?
* Is *Puppeteer* really necessary here?
**.gitlab-ci.yml**
image: cypress/browsers:node14.15.0-chrome86-ff82
stages:
- compare
compare:
stage: compare
script:
# install required dependencies
- npm install -g http-server puppeteer lighthouse@6.5.0
# check the current branch score
- http-server . &
- sleep 5
- lighthouse http://localhost:8080 --output=json --output-path=./current-branch-score.json --chrome-flags="--headless --no-sandbox" || exit 1
- CURRENT_BRANCH_SCORE=$(node -e "const data = require('./current-branch-score.json'); console.log(Math.round(data.categories.accessibility.score * 100));") || exit 1
# check the main branch score
- git remote set-branches --add origin main
- git fetch
- git checkout main
- http-server . &
- sleep 5
- lighthouse http://localhost:8080 --output=json --output-path=./main-branch-score.json --chrome-flags="--headless --no-sandbox" || exit 1
- MAIN_BRANCH_SCORE=$(node -e "const data = require('./main-branch-score.json'); console.log(Math.round(data.categories.accessibility.score * 100));") || exit 1
# logging out the scores and comapre them
- echo "Main branch score:$MAIN_BRANCH_SCORE"
- echo "Current branch score:$CURRENT_BRANCH_SCORE"
- |
if [ "$CURRENT_BRANCH_SCORE" -lt "$MAIN_BRANCH_SCORE" ]; then
echo "Current branch score ($CURRENT_BRANCH_SCORE) is lower than main branch score ($MAIN_BRANCH_SCORE)"
exit 1
else
echo "Current branch score ($CURRENT_BRANCH_SCORE) is higher than or equal to main branch score ($MAIN_BRANCH_SCORE)"
fi
# Add a comment to the merge request
- |
curl --location --request POST "https://gitlab.com/api/v4/projects/$CI_PROJECT_ID/merge_requests/$CI_MERGE_REQUEST_IID/notes" --header "PRIVATE-TOKEN: $PAT" --header "Content-Type: application/json" --data-raw "{ \"body\": \"🎉Lighthouse scores comparison:\n\nMain branch score: ${MAIN_BRANCH_SCORE}\n\nCurrent branch score: ${CURRENT_BRANCH_SCORE}\" }"
rules:
- when: on_success
**Thanks in advance. I listen to suggestions!**
https://redd.it/1c9hy07
@r_devops
How long to create a deployment pipeline
I recently had to create a deployment pipeline for a azure logic app.
The developers use a sandbox to to develop the app using the logic app GUI.
I needed to create a pipeline that takes the logic from a specific version of the playground logical app (for easy reversion) and moves into dev, and then dev to prod.
It took me like 12 hours to get all the azure prices lined up extracting the parameters from the repo for each instance, and pulling the logic from the logic app version and combining it all to make a template file that az cli can use to update the logic apps.
I ended up running like 100 test pr's just to get the git hub action working correctly.
Is this a reasonable amount of time for never touching a logic app before, or am I just bad at this?
Also I hate logicApps, they suck, seriously it's like 45 steps to do some basic stuff, that python can do in like 25 lines of code. I get that it makes it easier for people to understand the logic, but at the same time it kind of doesn't, because over time with edge cases I can see this logic app getting super complicated and no good way to handle logging (maybe azure analytics)? But that didn't even see straight forward compared to log analytics on an azure function, which is super easy.
https://redd.it/1c96w5z
@r_devops
How do you submit your expenses?
For those of you who need to submit your internet/phone bill each month, whats the process look like for you? do you manually download and upload the receipts?
https://redd.it/1c8xicd
@r_devops
Gitlab vs GibHub vs Bamboo+Bitbucket
Okay guys, I need your group wisdom. And knowledge.
I have a group of developers I need to support. Some of them are currently using GitLab and others Bamboo/Bitbucket. They, of course, prefer their tool. Still others are lobbying for GitHub. Due to company policies, we can't use AI, and whichever we choose will be on prem.
We also have a requirement to use GitLab in one of our environments, so my thought is any not standardize on GitLab? Then we only have one too to support and one type of yaml to maintain. We also have a sister company using GitLab that could be useful as consultants for GitLab. The people championing the other two are quite loud though. Also, the quotes have come in where the Atlassian tools and GitHub are much cheaper and about the same price... But GitLab is much more expensive.
I really think it should be GitLab, then if we don't get that, then GitHub, with Atlassian tools being a distant third choice.
We do use Jira and Confluence, but I don't think the integration with those is worth it. Also, we aren't sure if we want Ultimate or Premium GitLab. Also not sure if the security scanning in Ultimate is a decider.
I need a "smoking gun" to prove my recommendations are correct, but I haven't a concrete reason for this, other than above. This is to convince the high up leadership hopefully to go GitLab and make it worth it for price.
https://redd.it/1c8tc1b
@r_devops
Tools to create diagrams/graphs with detail view function
I'm looking for a tool to create a network graph with a "detail view" function.
E.g. something like Python diagrams or Mermaid that allows you to define a diagram/graph in code (with custom SVG icons) but with the ability to output multiple SVGs (because of scalability) and a display tool with zoom in function which allows you to get a more detailed view of parts of the diagram/graph that can be embedded, e. g. in a Sphinx documentation.
The idea is to visualize an entire network in one place with details only being loaded in on demand if that makes sense.
An example of what I imagine: Starting out you see a cluster overview, with just load balancing, frontend, API, DB and GlusterFS data clusters. When you click on the data cluster you see the bricks. When you click on the bricks you see the individual servers. And when you click on the server you see its hardware details. And of course there needs to be a "back" button that zooms out again.
Does something like that exist?
https://redd.it/1c8m8b0
@r_devops
GitOps Repos Structure
Hi,
I have question regarding the best structure for the GitOps architecture.
1. We want to manage K8s configuration with GitOps
2. We want to deploy all of our 70 small applications via GitOps to single GKE cluster
3. We do not use Kustomization, only simple manifest since our Apps out quite simple.
4. Each app has its own separate Git Repo for source code.
5. We have two environments (dev/prod)
Guys, what is the best approach for setting up GitOps for it?
1. Monorepo for all GitOps?
- two folders for env (dev/prod)
- each separate folder for specific app
- root folder for common k8s configuration (like, quotas, rbac, etc)
Thanks to that I can only setup one GitSource for the whole cluster on /dev env. But I will have around 120 folders in the GitOps monorepo.
2. Place folder for GitOps in the App GitOps Source Code where people will publish theirs Manifests?
.. what else can be done?
Thanks :)
https://redd.it/1caium0
@r_devops
Need suggestion!
Hi everyone, I have 2 yoe , currently in support. I want to move to Devops. Please guide me. I know the roadmap and basic docker, K8s. I am planning to take coaching either in Bangalore or hyd. Please suggest me... I tried learning from YouTube but I don't feel that confident.
https://redd.it/1cae2w1
@r_devops
Cross Account Image pull from ECR to EKS
I have 2 AWS accounts, let's say Account A and Account B. Account A has around 200 Private ECR repos and Account B has an EKS cluster.
I am trying to pull images from A to B in EKS.
I have tested the following for 1 repo and it works:
Added an IAM policy to the EKS Nodegroup role to get images from account A.
Created a Resource policy on 1 of the 200 ECR repos to allow EKS Nodegroup role arn.
But, the problem is if I go with this approach, then I will have to create the same resource policy on all 200 ECR repos.
Is there a better way to do it?
Thanks in Advance!
https://redd.it/1ca9xv6
@r_devops
90% of Java Services in Production Have Vulnerability Risk, DevSecOps Report Finds
Datadog’s State of DevSecOps 2024 report details Java service vulnerabilities and security scan noise.
With Java services being heavily targeted by attackers, should enterprises reconsider their language choices for critical applications, or should they focus more on strengthening security practices?
https://redd.it/1ca7cpz
@r_devops
Bitnami Helm Charts are Now More Secure Than Ever
https://tanzu.vmware.com/content/blog/bitnami-helm-charts-are-now-more-secure
https://redd.it/1ca5uhn
@r_devops
Seeking For Career Guidance
Hello everyone,
I am a student pursuing my bachelors, I have a keen interest in Cloud and DevOps and I wish to pursue my career in it. My skill set includes :
Cloud Platforms: Amazon Web Services (AWS)
Containerization: Docker, Kubernetes, Vagrant
Continuous Integration/Continuous Deployment (CI/CD): Azure DevOps, Jenkins, Github Actions
Scripting: Python, Bash
Programming Languages: C, C++, Python
Version Control: Git
IAC: Terraform
Operating Systems: Linux (Ubuntu, CentOS)
Monitoring: Graphana, Prometheus
I also have an AWS Certified Cloud Practitioner Certification. I am confused right now what should I do next. Should I pursue more certifications, or build project(If this then please suggest some ideas or resources) or should do Internships(I wish to do so but I am unable to find could you please give insights on how can I find internships other than using portals like LinkedIn, internshala or wellfound should I dm to the company directly attaching my resume ?) or should I do freelancing (in this please guide me on how to get clients and grow Im stuck at it too) or should I learn some new skills.
How can I know how much is enough for me am I ready for internships or not cause Im constantly learning and using the technologies....
Any guidance is appreciated.
Thank you
https://redd.it/1ca3abp
@r_devops
I'm building a service to provide insight into last mile network errors.
Hello, fellow engineers! I am building CiThru, an observability service aimed at identifying user access issues to websites, which are usually a blind spot for traditional monitoring. The goal is to build a system that can simplify the process of diagnosing network issues that traditional metrics do not reveal, such as ISP-related blocks, issues with firewalls, and CDN outages.
There is a bit more detail on the idea behind the initial implementation in this blog, https://www.cithru.com/blog/nel/, but the initial implementation heavily relies on the Network Error Logging (NEL) browser spec—once you open a website, it sends back a header with a special URL and your browser remembers it. Next time there is a network error opening this website, the browser will report it to that URL. There is also an amazing talk on this technology here: https://www.youtube.com/watch?v=x96nzfxGJ4I. I’ve personally used NEL to debug and investigate various network & ISP-related outages, and I hope the tool I’m building can be useful to others.
CiThru is in early beta. If you're interested in trying it, please let me know and I'll send you an invite. I'm very keen to learn whether it's something that others can find useful. Thank you.
https://redd.it/1c9zyhd
@r_devops
Given this tech stack, would you say this is on cloud or on-prem?
Let's say a company has architecture represented by Linux, Oracle, OpenShift, Hadoop, SQL Server, MongoDB.. would you say this is on cloud like AWS or on site?
Also two questions related to this
1. Is it feasible to implement such an architecture on the cloud?
2. If so, why would a company choose to do this, given that people on AWS might typically opt for AWS native services? I'm currently learning DevOps, so I thought I'd ask here. Thank you! :)
https://redd.it/1c9t2k7
@r_devops
Is there a middle ground somewhere between Docker and Kubernetes?
Hi guys,
I'm not a professional developer, but I'm picking up some skills to try bootstrap an open source data liberation project that is stoking my passion.
The open source data platforms I'm looking at using for my own project mostly required either Docker or Kubernetes so I had to play around with both of them. Docker ... I found pretty easy to get going on a Linux VPS, although the internal networking sometimes tripped me up.
Kubernetes is amazing but ... super hard. The only thing I was able to get working was a GKE image and I feel like I cheated!
The applications I'm playing around with are mostly two or three containers that typically come packaged in a Docker compose. I'm not expecting any major traffic in the short term so the whole Kubernetes framework seems way overkill for my needs.
Equally there's something nice about the way it puts things together in a package. For most of my projects I need a database (usually PostgreSQL) and then an app (right now I'm playing around with Apache Superset, Metabase, and Grafana).
Is there any cloud native platform that's kind of geared towards taking a Docker compose and patching together a little environment? A halfway house between the two as such?
TIA!
https://redd.it/1c9pc5p
@r_devops
what is the difference betwween cloud and devops
and what's the best way to learn each one pls
​
https://redd.it/1c9mpp3
@r_devops
Cheapest Cloud Servers out there?
Hello everyone!
I am currently searching for a cheap cloud servers and i have come over some options, like AWS EC2 or services like Kamatera or HostGator. However, while they all are relatively cheap (Like $7 or so), they also just provide 2-4GB of RAM. Is there a lesser known service that is as cheap as $5/Month with \~10GB RAM, a decend enough CPU and network connection to host web- and game servers on?
https://redd.it/1c9i15b
@r_devops
Do large companies typically use a managed Kubernetes or self-host?
Hey,
I'm pretty familiar with managed Kubernetes on two cloud providers. Azure & AWS. Deciding whether to spend a weekend spinning up a self-hosted k8s to deepen my knowledge [ Trade off between doing some programming or this \]. Do large companies self-host their k8s, or does everyone typically use managed for production nowadays?
Cheers!
https://redd.it/1c93mv6
@r_devops
Overqualified and Underpaid
Last year, after 6 months of interviewing, I accepted the only job offer I received. It wasn’t a Senior-level position and I took a 35% pay cut. The expectation was that I would come in, show my worth, and be promoted to a Senior title and a decent pay raise.
After completing my onboarding in 2 months instead of 3, I’ve been given the responsibilities of a Senior DevOps Engineer and excelled. It’s been almost a year and I have exceeded everyone’s expectations.
Despite everything, management continues to drag their feet on promoting me and I’ve lost confidence that they will until maybe early next year.
I recently started looking for a new job with a Senior title and pay. The number of job postings seem slightly better than last year, but salaries are lower.
I’m curious if others are in the same boat and wondering how they are approaching their job search.
https://redd.it/1c8v0vx
@r_devops
Senior Dev to DevOps transition
Wanted to ask what skills I should read up on before switching to DevOps. My current and prior companies has had massive problems finding DevOps people and I know the wages are higher. So been thinking about changing teams.
I think it's mainly imposter syndrom holding me back. I have 15 years of experience in software development. I have worked in both Azure and AWS cloud. Had hobby projects hosted in both.
I am currently hosting my own K8's cluster on Hetzner ARM instances for my private projects. Running postgres, REDIS and different WebApps / apis aswell.
So I would say I have done a wide range of DevOps tasks in prior jobs setting up and maintaining build pipelines in Azure and GitHub.
What I mainly lack is "real" production hosting of databases and backup strategies, since it's either been handled by others or not caring losing data in hobby projects.
I am Abit lost what to read up on before applying for team change. Any good advice?
https://redd.it/1c8of46
@r_devops
How to deploy a multi-service application
I have a backend that looks like this
https://excalidraw.com/#json=HF2\_DBOgoti6C6tbVG\_6k,ZXerWMQn1sKcnySy3hUWFQ
I think a lot of people have similar architecture, but I have not seen a shared common solution online for such architecture
locally I use docker-compose, but from what I understand it's not recommended for prodcution(not sure of this)
How to deploy this app architecture to production(without kubernetes)?
What tools people use?
https://redd.it/1c8eglj
@r_devops