DevOps and other issues by Yurii Rochniak (@grem1in) - SRE @ Preply && Maksym Vlasov (@MaxymVlasov) - Engineer @ Star. Opinions on our own. We do not post ads including event announcements. Please, do not bother us with such requests!
“5 Whys” is a fairly popular framework for searching for a contributing factors of an incident.
However, this framework has its own limitations. Here is an article that presents these limitations in a concise way.
#sre
For today's Donations Monday we have a fundraiser from Dzyga's Paw that they do together with 7 different units. My friend's brother serves in one of those units.
https://send.monobank.ua/jar/7CRy1e16Qk
Here's the description from Dzyga's Paw themselves.
Dzyga’s Paw Fund, in partnership with seven units, is launching a $300,000 fundraiser to provide them with 90 night drones — essential for precision and safety in night operations.
Donate now to support project Triad: https://dzygaspaw.com/triad-night-drones
We are bringing together two powerful forces: our international supporters and Ukrainians backing their brigades. No matter what they say, Ukraine is not tired. Our defenders fight on, and we stand with them. And our friends from all over the world are eager to help us with this mission.
Join our ambassador team to help reach this ambitious goal faster — start a smaller fundraiser! Write to us in DM, and we will provide you with all the information, visuals, and donation chart.
This war has lasted nearly 11 years, and the last three have been the most brutal. Against all odds, Ukraine continues to resist. It is our duty to ensure our defenders have the tools they need to win!
#donations #Ukraine
Some books on programming and software architecture on Humble Bundle.
#books #programming
Last week, we closed Maksym's jar for Hospitallers in a day! Let's now do the same favor to another ember of our community - Dmytro!
His wife is raising funds to repair two cars for the 41st separate mechanized brigade. The goal of the support jar is 30k UAH. Also, there's a raffle for all the donations which are multiples of 100 UAH.
More info about the raffle is in this Instagram post (in Ukrainian):
https://www.instagram.com/p/DHV5cfCg1GU/
#donations #Ukraine
AWS gives a chance to complete some certifications for free, but with few preconditions
Covered certs:
- Foundational: Cloud Practitioner, AI Practitioner
- Associate: Solutions Architect, SysOps Administrator, Developer, Data Engineer, Machine Learning Engineer
Note, that you need to prepare and pass exam(s) before Aug 2025.
#aws #certification
Today, I'd like to share with you a nice YouTube channel Polylog, that does short explainer videos about the computer science concepts.
PolylogCS" rel="nofollow">https://www.youtube.com/@PolylogCS
#programming #cs #youtube
The highest priority after completing the task is to preserve the life and health of the personnel. So, let's help raise money for tactical medicine!
TL;DR: Donate to Monobank jar till EOW and I will double the amount raised[1].
The Hospitallers Battalion is a volunteer formation known for its professionalism and dedication. When I say "voluntarily," I mean that you can join for a rotation, for example, for 2 weeks, and then return to civilian life for a few months before repeating the process. This flexibility allows specialists who, for various reasons, do not want or cannot join AFU still contribute to defensive operations. However, since the Hospitallers are a volunteer formation, their funding relies on donations from ordinary people.
Currently, the Hospitallers are raising 7.5 million UAH (~$180k) for this spring.
Let's aim to raise at least 10k UAH from CatOps by the end of the week, and I will double the amount raised[1].
Monobank jar: https://send.monobank.ua/jar/6eEHjgDTGq
Btw, the Hospitallers offer courses on first aid and tactical medicine that are available to civilians. I recently took the TCCC ASM (3-day) course, and it was excellent. You can check out their courses at Hospitallers Courses site and track new dates on their Telegram channel.
#donations #Ukraine
[1] But not more than 20k UAH/month from me until the fund campaign is closed or until I have doubled the amount, whichever comes first.
Yet another article from the old archives.
This one is about building a multicluster setup with Kubernetes using Kubernetes tools only. Moreover, there are more tools these days that can help you accomplish the architecture outlined in this article.
For example, you could use ClusterAPI instead of Crossplane for leaf-clusters and so on.
The only sad thing is that Kubernetes doesn’t support federation. So, any viable multi-cluster setup always boils down to two distinct approaches: a bunch of independent clusters, or a single leader with leaves.
#kubernetes
How to Scale Elasticsearch to Solve Your Scalability Issues is a neat guide for areas where you could optimize your ElasticSearch clusters.
Ties article doesn’t provide any concrete numbers, but gives plenty of suggestions. It makes sense, since any optimization heavily depends on one’s use case.
#elasticsearch
The first Spring issue of the newsletter!
https://newsletter.catops.dev/p/catops-digest-2025-03-09
#digest #newsletter
Another article from the old stockpiles, and again it's from Julia Evans.
This one is about the ways DNS can break.
It's been a couple of years since this article was published, so some things may be irrelevant today. For example, the issue with DNS over TCP was fixed in musl
to my knowledge.
In any case though, this is a nice article that deserves your attention.
#dns #networking
How Do Websockets Work is a short explainer of websockets by the System Design Newsletter.
BTW, that is a good newsletter, I can totally recommend!
#web
From time to time I’m revisiting old articles that I have saved somewhere or those that just laying around, etc.
Today, I’d like to share with you two articles by Julia Evans on how to learn new skills, which is a skill on its own.
The second article is actually a presentation from the Strangeloop conference on how to reason about things and make hard or confusing things easier to understand.
#education
Terraform Patterns, Observed. Part 5: Logic in Terraform is a nice article with observations of how people incorporate logic in their Terraform code, even though TF’s logic is somewhat limited.
BTW, as you can see, this is a part 5. Robert has other articles with Terraform observations on his Medium. I highly recommend checking them out as well!
P.S. This article is brought to you (and me) by weekly.tf - a newsletter about the Terraform ecosystem by Anton Babenko. This is a great place to keep up-to-date with Terraform, even if you don’t work with it a lot.
#terraform
On Saturday, I gave a talk at the FW Days DevOps conference. I will share the videos with you once they are available.
During the conference, they were raising money for a night vision device PVS-14 for the 4th battalion of the 46th separate brigade.
The goal is to raise 220k UAH, and almost 150k was raised during the conference. Let's help them to close this goal!
Monobank Jar: https://send.monobank.ua/jar/vxr5ywPjM
#donations #Ukraine
Lurking Variables is a story about contributing factors. Specifically, those factors that people do not take into account right away, or better said, are not instinctively taken into account.
Anyway, this is a good read about a very real problem of confirmation bias when handling incidents.
#sre #incidents
A new issue of the CatOps Digest is here:
https://newsletter.catops.dev/p/catops-digest-2025-03-30
#digest #newsletter
We are continuing our security marathon with some news about very popular NGINX Ingress for Kubernetes.
Wiz Research discovered CVE-2025-1097, CVE-2025-1098, CVE-2025-24514 and CVE-2025-1974, a series of unauthenticated Remote Code Execution vulnerabilities in Ingress NGINX Controller for Kubernetes.
Exploiting these vulnerabilities can lead to unauthorized access to cluster secrets as well as remote code execution inside the ingress pod.
This vulnerability is fixed in Ingress NGINX Controller version 1.12.1 and 1.11.5, but if you cannot upgrade right now for whatever reason, Wiz has other recommendations on how to mitigate this.
P.S. Many thanks to the chat for sharing this story!
#security #kubernetes #nginx
A new vulnerability is discovered in Next.js - a popular web framework based on React. This vulnerability allows an attacker to bypass the redirect middleware as well as potentially make a cache poisoning attack.
Here’s also a thread by Gergely Orosz about this issue.
#security
A sneak peek into the database design based on two incredible books: "Database Internals" by Alex Petrov and "Designing Data-Intensive Applications" by Martin Kleppmann. Mostly on the first one, though.
This article touches topics of ACID, underlaying data structures (B- & LSM-trees), and distributed systems. Just like the aforementioned books, which I also highly recommend!
A random quote:
> Going distributed should be a last mile resort, introducing it to a system adds a ton of complexity, as we will soon learn. Please avoid using distributed systems when non distributed solutions suffice.
#databases
Python is incredibly popular programming language among DevOps / Site Reliability / Platform specialists. It’s also used as the language of choice to build backend in many companies.
So, here is a book bundle by Packt to enhance your Python skills:
https://www.humblebundle.com/books/python-from-beginner-to-advanced-packt-books
P.S. I know that many folks are skeptical about Packt books, so keep this in mind. I still think it worth sharing, though.
There are two types of folks: those who pin their dependencies to (often) mutable tags and other pointers, and those who already pin to hash sums.
This is true for GitHub Actions as well. For example, most tags in the tj-actions/changed-files repository were repointed to compromised versions by an attacker, potentially affecting 23,000 repositories until GitHub shut down the entire repository.
Manually pinning dependencies can be tedious, so it's better to utilize tools like Renovate. The SpotOnInc/renovate-config preset does exactly that. Note that you can pin it to a hash sum too, just in case ;)
P.S. If you used that GitHub Action, here is a list of possible mitigation steps: antonbabenko/pre-commit-terraform#837.
#security #github #gha
Git is one of those things, where you can discover new things even after decades of using it.
This article provides some tricks you could do with Git. If you knew those already - that's cool. If not, you're welcome :)
#git
I do not have small fundraisers for you today, so let's donate to Serhii Sternenko for FPV drones:
https://send.monobank.ua/jar/2JbpBYkhMv
#donations #Ukraine
A book bundle on various computer science topics by No Starch Press.
https://www.humblebundle.com/books/computer-science-fun-way-no-starch-books
#books #bundle
One of the interesting questions when it comes to SLOs is how to define them, in other words, how to set up boundaries to make the SLOs meaningful.
This article by Honeycomb provides some ideas on how to organize your SLOs, so on one hand they represent the user's needs, but on another hand do not cover things that your team cannot control.
#observability #slo #sre
Let’s help Seva to close this fundraising round for a night vision device!
We could close it right away, if only every subscriber of this channel sent ₴6 there.
Monobank jar:
https://send.monobank.ua/jar/vxr5ywPjM
#donations #Ukraine
For my DevEx specialists out there: What's Next in Measuring Dev Productivity?.
tl;dr: this is kinda a very basic description, but at least it has a table with metrics and some justification behind them. This is why I decided to share it.
#culture #devex
What is common between Helm and the article about the abuse of the curl | bash
setup to load bad payload?
Year of initial release. But if the article is not more accessible outside the Web Archive, Helm still provides that as a valid installation option.
Anyway, the article is old but gold, worth reading to understand how tricky attakers can be.
#security #bash
CatOps Digest 2024-02-23 is here!
https://newsletter.catops.dev/p/catops-digest-2024-02-23
#digest #newsletter