86
Reddit DevOps. #devops Thanks @reddit2telegram and @r_channels
Who's hiring right now? (US markets)
Been out of market for sometime and applied few roles. But havent heard back from recruiting team yet.
Want to find out how the current DevOps market is right now? I know as a whole things are really slow, but some development roles are opening up a bit around the country.
🙋♂️
https://redd.it/1d0dp6d
@r_devops
Read-only filesystem for Docker
Hi!
I'm a DevOps Engineer and my company that I'm working for right now is working on an embedded system where our apps need to be deployed using Docker.
Our single board computer has two memories:
- 1 x memory soldered on the board - 1 GiB (system),
- 1 x eMMC memory- 256 GiB (in fact: 25 GiB partition for Docker apps).
I'm trying to figure out a way where our apps are run in a read-only mode. However, I don't mean the read-only mode for Docker containers (i.e. read-only option in Docker Compose or --read-only via Docker run command), but the whole Docker Engine.
The problem:
Our eMMC memory has 256 GiB. The memory wears out very quickly if there're a lot of write operations.
The goal:
We'd like to limit write operations on the disk, so it can last for years. The system will be used in ambulances in very heavy conditions; constant hard power-offs, high temperature, high humidity and other factors will happen quite often.
My work so far:
Docker uses /var/lib/docker directory for all of its data. There're sub-directories inside:
# Directories That Can Be Read-Only
1. `buildkit/`:
* Generally, BuildKit configurations and some cache data. Can be set to read-only if no active builds are happening.
2. `containerd/`:
* Contains containerd runtime data. Can be set to read-only if the container runtime is stable and no new containers are being created frequently.
3. `image/`:
* Stores Docker images and metadata. If you are not pulling new images frequently, this can be read-only.
4. `network/`:
* Contains network configurations. If your network setup is static, this can be read-only.
5. `plugins/`:
* Stores Docker plugins. If you are not adding or updating plugins, this can be read-only.
6. `runtimes/`:
* Contains runtime configurations. If your runtime setup is stable, this can be read-only.
7. `swarm/`:
* Stores Docker Swarm data. If you are not actively managing a Swarm, this can be read-only.
# Directories That Need to Be Read-Write
1. `containers/`:
* Contains running container data, logs, and state files. Needs to be read-write.
2. `overlay2/` (or other storage driver directory):
* Stores the writable layers of your containers. Needs to be read-write.
3. `volumes/`:
* Contains data volumes used by your containers. Needs to be read-write.
4. `tmp/`:
* Temporary files used by Docker. Needs to be read-write.
5. `engine-id`:
* Stores the engine ID. Needs to be read-write initially, but could potentially be read-only after setup.
6. `lost+found/`:
* Standard directory for recovered files, typically read-only but can be left as-is.
The question:
The question is: how to set Docker Engine to be read-only for all the time and set it to read-write when needed (e.g. firmware updates)? I was trying to remove subdirectories that need to be read-write and put there symbolic links to /tmp directory (so it is run in RAM memory). We just want the eMMC memory to last for years. Any ideas what should I do, consider?
https://redd.it/1d00h0q
@r_devops
Brisk, a complete system for running CICD is now open source
Brisk is a tool for running Continuous Integration tests, think CircleCI, but much faster.
Instead of building the test environments on each run we pre-build the environment so that your tests run as fast as possible. Combined with enough workers this gets test suite run times down to seconds (https://brisktest.com/demos).
I've had a lot of requests for people who want to run Brisk in their own infrastructure and so I've gone ahead and open sourced it all. You can now run Brisk yourself both locally (for development/testing purposes) and in your own cloud for free.
Currently we have dedicated support for Node JS, Ruby, Python and Go test suites and we integrate with Github Actions, CircleCI etc, Bitbucket etc. Adding additional test runners for other languages is also straightforward.
We also integrate with your IDE so you can access the full power of your CICD pipeline from your local machine after each save.
Let me know what you think.
https://github.com/brisktest/brisk
https://docs.brisktest.com/
https://redd.it/1czxeko
@r_devops
Installing and Deploying Docker with cloud-init
Is there a recommended approach to installing and deploying containers on a Debian VM with cloud-init?
I found some answers on StackOverflow but I wanted to be sure if it was the recommended way. TIA!
https://redd.it/1czutr9
@r_devops
Christophe Limpalair Cheat Sheet on Excessive Priv + Zombie Cloud Identities
Nice vid explanation in undr 180 secs. https://www.youtube.com/watch?v=lyPXDCbfdR8
https://redd.it/1czm7rf
@r_devops
CKA or CKAD?
Hey people.
Senior SRE here with several years of experience developing and managing k8s clusters. I've been studying for the CKA cert, however after thinking about it some more, CKAD is starting to make more sense to me.
The continuous trend in k8s administration is through managed services like EKS and GKE. Rarely do I see job openings of bare metal or administrating a cluster through other means. While yes there are still cases where you need to debug the control plane, it's extremely rare and 99% of the time, the associated cloud provider has support.
And yes, it's extremely important to know how the control plane component works. But, I find that my personal experience with K8s is more in the development and application architectural side, than it is in debugging etcd or the api server.
Does anyone else feel the same? Or is CKA still the recommended cert to get?
https://redd.it/1czpxbj
@r_devops
Pagerduty vs. OpsGenie vs. VictorOps vs. Grafana ... what's the deal?
Hey everyone. I work for a large company that is a household name, but we are not really a tech company and our stack is woefully outdated. We're finally getting the budget to modernize our ops and devops now in the wake of the covid disruption.
Like everyone else, we already use atlassian. We happen to use splunk as well. We're in an RFP between pagerduty, victorops (who I believe is the splunk one) and opsgenie (who is the atlassian one). We are also looking at grafana.
My main question is ... is a stand-alone incident response platform even justified these days? Seems like the good enough offerings from splunk and atlassian are the future and xmatters and pagerduty are kinda going to lose relevance over time.
Anyone else feel this way or are you seeing something that I'm missing? Why should we even consider pagerduty when it seems the others are closing the gap. I get that pagerduty seems to be the most fully featured, but we're a large org and don't need half the stuff they're offering. Like killing a fly with a bazooka it seems.
Is there some critical feature or functionality pagerduty has that victorops/opsgenie/grafana don't have that are causing orgs to continue using pagerduty instead of consolidating into atlassian or splunk?
Cheers!
View Poll
https://redd.it/1czmxr4
@r_devops
Manually assigned on-call status
In a previous job as a developer I had a share of an on-call rota. The rota was managed by PagerDuty which would automatically put me on call when it was my turn.
That generally worked fine, but there were a few times when I missed the notification and didn't realise when I was on call.
So I thought a much more reliable system would be one where people have to go on-call manually, and the system generates an alert to the team if someone isn't on-call when they're supposed to be, or if there count of people on call at any time is too low.
Does anyone know if pagerduty or any of its competitors support a system like that?
One person I know did on call manually doing IT in a medical environment, but I think that was entirely manual, with nothing to route alerts to the on-call dev, devs just declared themselves on call on a chat channel or something like that and then watched for alerts.
https://redd.it/1czjzgq
@r_devops
Need help in building a project
Hii eveyone, i need some suggestion in a project i am building, the project is a leetcode clone i need for my company, i want to execute code and run it against test cases, so i am using django for the api and dockey py to talk to docker API. i have some approaches to how to execute code, like i can run a container everytime i need to execute a code and then delete it, but it takes time to spin up a container. next is to used pre run containers, so i need suggestion on that and how can i run testcases efficiently, like i have 100 testcases for a code how can i run it again the given code, do i run it in a single container, or do it run it parallel on many containes, if parallel how can i manage those, etc. It would be helpful if someone is willing to have a chat
Thank You
https://redd.it/1czhc7n
@r_devops
What do you think about the future of terraform ?
After IBM take over hashicorp ,what is the future of their products?
https://redd.it/1cz7hcb
@r_devops
Docker to kubernetes advice
My company are moving from a docker container architecture hosted in ec2 instances to a kubernetes managed clusters. Whilst also investigating EKS and if if it suits our needs, I plan on bringing up the existing containers in a kubernetes cluster. What is the general concensus these days on the best option/s for doing this? I have played about with Minikube and microk8s in the past but trying to figure out what the modern state of the art is without stumbling through lots of bad solutions first. Any advice welcome :)
https://redd.it/1cz4ivl
@r_devops
Octopus Deploy new 10x prices are just ridiculous, what are our options?
We were aiming to renew out CI/CD platform license when we got hit with the new May 1st prices https://octopus.com/pricing/overview . Our env is not even that massive, we have about 100 endpoints and a couple dozen projects. It went from a couple thousand dollars for a perpetual license a few years back to now 20-30k with a not much bigger installation. Is there anyone else here with my on the same boat? what have been your options and what are your take aways so far?
We have spent years building our projects in Octopus and it will be a pain to uproot but we are measuring up out costs vs time. Azure DevOps no longer feels like a bad move but wondering if there are free options that could compare. I am aware of https://www.fdeploy.com/ project which is a bit of middle finger to Octopus for what they did but I have very low hopes in that ever being a thing, its not even ready to play test.
https://redd.it/1cz0te4
@r_devops
Good approaches to intentionally inducing errors in application/infra code?
I'm working on new logging and maintenance services to capture errors in our org's dev environment, and propose fixes based on the error type. e.g. a containerized application on Azure hits a runtime error and logs it. One of the new services grabs the logs from App Insights and processes the error, looking for misconfigurations in the app.
The problem I'm having is that, to scope and build this properly, I need to induce a wide variety of application- and infra-level errors in our dev env, i.e., an approach to breaking app/infra code in well-defined, ideally programmatic ways. Like chaos testing, but no randomness.
I don't even know how to start thinking about this. Have you had to intentionally break a broad set of apps before, to test your maintenance process? Are there mental models and processes you use to keep this organized? Is there tooling that can help?
https://redd.it/1cyxded
@r_devops
Automate documentation website update
If you are a developer maintaining documentation website with tools like Hugo or Docusaurus , I am curious to know the level of update automation you have in place and how you handle project release notes.
Meanwhile I am sharing a little experimentation https://www.updatecli.io/blog/welcome-releasepost/
https://redd.it/1cyvd8u
@r_devops
What are you using to create/manage/auth api keys?
I need to implement some type of API Key system to enable 3rd parties (paying customers) to use apis. I found unkey but not much else in this space, which is kind of surprising. Is everyone really rolling their own solution for this?
https://redd.it/1cyri37
@r_devops
GitOps: common practice merging into main/master?
scenario:
1)starting merging components into prod from staging/qa branch
2) discovered conflict in one of the components in prod > stging.
3) reverted the merged branches to not break prod.
4) revert doesn't exist on github portal, need to create a new branch based on previous commit-then push it onto dev/stg, to be merged into prod. since prod is protected and i couldn't push a commit into.
5) then after conflicts resolved, could merge sqaush from stg to prod with the newest version.
now, I'm guessing what i should have done(junior here, first time doing this), to avoid the revert part, is just using the drafted/review of PR(pull request), on all components to see if there are conflicts, and then I didn't need to revert at all.
any critical stuff i should avoid?
also is #4 common practice in such cases of protected master?
general critic is welcome.
https://redd.it/1d07orc
@r_devops
Guaranteed I’m making this way harder than I need to.
So my company has been running the kube-prom-grafana stack with the Prometheus-community helm charts. I’ve been asked to look at adding Thanos to the stack as a receiver.
Thanos.io docs on the non-sidecar setup are abysmal. Lots of dead links and a general lack of good info on that setup. Everything is geared towards running Thanos as a sidecar to Prometheus.
Anton Putra (YouTube) has some decent demos.
But I haven’t been able to get my head around it & I think I’m probably thinking too hard about it & missing the Forrest for the trees.
Does anyone have any recommendations on that setup?
Side note: Anton Putra has some excellent videos on YouTube covering a broad scope of technologies & his git repo is very well organized. Worth a look
antonputra?si=CfnTfAupGr3xEvgo" rel="nofollow">https://youtube.com/@antonputra?si=CfnTfAupGr3xEvgo
https://redd.it/1czxr2k
@r_devops
Fun Udemy courses?
I'm looking for a fun and instructive udemy course. Doesn't have to be strictly DevOps but I'm looking to spend some free time on learning something new that doesn't just have to be strictly related to my current job.
Did you recently do any Udemy courses that you really enjoyed?
https://redd.it/1czwxm4
@r_devops
You go back 5 years and can ask your younger self to learn one skill that's relevant today. What would it be?
For me, I'd learn some platform engineering concepts
https://redd.it/1cztr6w
@r_devops
Does canary deployments save your ass?
I have never been in an infrastructure with canary deployments. So I would like to know how well they work for you. How often do you acturally rollback a canary? How difficult is it for you to design services that needs to work in the mixed state? What situations have you experienced where canary deployments acturally saved your ass?
https://redd.it/1czoydz
@r_devops
Advice on cloud infrastructure setup (multiple single servers vs cluster of servers containerized)
I am building a few apps currently and want to deploy each app soon to production, still thinking on how to setup the infra.
Most of the apps will be a node web api and a database (mongodb / postgres) for multiple clients or my own apps. I am a one man army and will invest everything myself so I am not on a huge budget currentlt.
First idea is:
Setting up a small server for the app instance and also a seperate small server for the database. This will be upgraded when needed. With this I dont have multiple database nodes for any failure fallback / replication strategy. And also not multiple servers for the app to have a proper load balancing + zero downtime strategy. Apps will be installed bare metal so that gains probably some performance over containerization (using nix, so everything will be declarative and reproducible).
Second idea:
Setting up a few big servers (more when needed) in a cluster and deploying every app and database instance in a container. This way I could have multiple container instances to allow replication and zero downtime and have them on seperated servers for fault tolerance. I will actively monitor and add more servers to the cluster or increase the servers when needed.
----
I do value a good infrastructure to also handle zero downtime and have some database replication happening for fault tolerance.
So i was thinking with the first approach it will become a huge pay check when I have multiple apps deployed because each app needs at least 2 servers and maybe more if i want to have replication and other good stuff. But I am a beginner so no idea if it actually going to be more expensive..
With the second approach I am curious whether it is recommended to have such an infrastructure for production apps and if it becomes a bottleneck to monitor all the containers on multiple servers...
Any advices or suggestions are much appreciated!
https://redd.it/1czoirq
@r_devops
Help needed for a usability test
Hey everyone, we are running a usability test for some improvements on the notifications of XWiki. If you have some time (it should take 5 min) and want to participate, please follow the link in this message. Your participation is anonymous and will be greatly appreciated. Thank you all! https://dev.quant-ux.com/#/test.html?h=a2aa10aDO5BZ3legXwvGxkQzrlN3uQhy1KpyJ4xsGaJOsnYzlYe5Efdi32Ty&ln=en
https://redd.it/1czlm3u
@r_devops
JCR vs Artifactory repo question
I have a dumb question, I got JCR running in my lab, and when I have created a repo, I can see the same SHA in my repo as when I do inspect on my local images. When I look in our work's proper Artifactory, I never seem to find the SHA's. Looks like I have the same images and tags, but I just can't see the SHA's in the GUI (yeah, I know I'm noob). Any enlightenment would be helpful!
https://redd.it/1czh9yf
@r_devops
How to master DevOps? (Like kubernetes)
I should say the demand for kubernetes is going day. Have been working in start up for around 2 years, unfortunately still doesn't use kubernetes actively. How should I develop my skills particularly in kubernetes and helm. Is completing a course from platforms like KodeKloud validates my skill on social media? I genuinely feel I'm stuck at my current company and wanna switch for more exposure. But this job market..... Aghhhh.... is not the only problem. The technical interviews, somehow, are always under the belt.
Any advice?
Please share your experience for starting as freshers in the field of DevOps.
https://redd.it/1czg7li
@r_devops
Why not have shared dev environments with traffic and data isolation?
At my previous job every DEV had their own dev environment which mimic'd our entire architecture. This feels prohibitively expensive to me, I am wondering how this would compare to an architecture where there was a shared environment with traffic and data isolation; so that my dev traffic didn't touch my colleagues dev traffic (if I choose for this to be the case).
https://redd.it/1cz6uj4
@r_devops
Streamline Development with Effective API Mocking Tips
Hey, hey just sharing some API mocking tips. Check out the below blog for an overview of: what is API mocking for the newbs, 7 advantages of API mocking, how to integrate your mock APIs with CI/CI pipelines, plus a few advanced mocking techniques.
Any advanced mocking techniques that we missed or should add?
https://www.getambassador.io/blog/streamline-development-effective-api-mocking
https://redd.it/1cz3jbu
@r_devops
Today I discovered Skopeo and it has changed my approach to a product I am working on.
Disclaimer: not a Red-Hat employee nor an OCI foundation member.
TL;DR: Skopeo is great for Air-Gapped Container Registry syncs and image management.
I am currently working in the Industrial IoT sector where a lot of Cloud-Native products are seeping in. The only issue is customers of Operations Technology despise Internet Connectivity. Without Internet Connectivity, working with Cloud-Native technologies like Containers is a massive pain.
So as a common way to solve such a problem I was working with Ansible to setup systems with a Docker Registry and then using container tarballs to do the following:
- docker pull images from registries
- docker save to create tarballs
- docker load to extract them on the main gateway
- docker tag to make sure I rename it for making it compliant with the registry
- docker push to registry
- cleanup etc.
This works well for a single architecture system e.g. amd64. Now I have devices which also are arm64 based and the registry isn't smart enough to generate manifests to let devices determine which blobs to pull based on the underlying architecture.
I tried Skopeo today and in the lab I was able to sync an Image from Docker Hub to a testing registry in 10s.. No docker gymnastics involved.
If i need to create tarballs - Skopeo can do that. If I need to push tarballs to a registry it can do that too. Cherry on Top, no need to have docker installed on your local machine where some development is needed.
I will combine Ansible and Skopeo to get a large leap on the product I am working on.
The only thing I need to figure out is to generate a Manifest List for multi-arch container images available on Docker Hub
Amazing tool, would recommend for air-gapped scenarios.
https://redd.it/1cyyxlm
@r_devops
Seeking Guidance and Mentorship for a DevOps Career Path
I am currently a BCA student entering my third year, and I have decided to embark on a journey into the world of DevOps, with the ultimate goal of pursuing my master's degree in Australia. I would greatly appreciate any guidance, resources, or advice on how to best navigate this path.
Furthermore, if anyone is willing to mentor me through this process, it would be a tremendous honor and deeply valued.
https://redd.it/1cyt4pf
@r_devops
Free tier of GitLab CI/CD
We are beginning to work in GitLab to create our pipeline and are working in the free tier of it.
Just got hit with the “namespace had 5% or less shared runner pipeline minutes. Once it runs out, no new jobs or pipelines will run.”
We are startup right now, and don’t have much of a budget.
I didn’t know how many minutes we needed, I thought that the free tier would be adequate enough.
Any advice ?
https://redd.it/1cysw32
@r_devops
How are you tracking / monitoring your SLO/SLIs
Im in AWS and would like to start adding some SLO/SLIs to out observability solutions. we primarily use cloudwatch logs / prometheus / open telementry for our metrics.
Looking around the only tools that i can see that work in all options is noble9 but that locks you into there dashboards etc, where all the rest of reporting is in grafana.
What tools and services is everyone using to monitor and track there SLO/SLIs?
https://redd.it/1cyogqc
@r_devops