5629
DevOps and other issues by Yurii Rochniak (@grem1in) - SRE @ Preply && Maksym Vlasov (@MaxymVlasov) - Engineer @ Star. Opinions on our own. We do not post ads including event announcements. Please, do not bother us with such requests!
So, that's for AI in the companies, but what about AI in the wild i.e. in open source?
We have cases like curl, that had to take down their bug bounty program due to the influx of slop bug reports. Yet, the industry adapts.
Here's a study by Redmonk on the stance of various foundations and standalone open source projects on AI, including their major concerns, and openness to AI-generated contributions.
#ai #open_source
Harness engineering for coding agent users is a new guest article in Martin Fowler's blog that summarizes approaches to improve AI output and make it more manageable.
If you're actively using AI agents day-to-day, things described in this article won't be news to you, but it helps to structure one's thoughts.
#ai
For today's Donations Monday, I'd like to share with you a fundraiser that our friends at DOU started for the 2nd separate corps of the National Guard of Ukraine «Хартія». The goal of this fundraiser is to buy heavy bomber drones "Vampire" for the Kupiansk direction.
Monobank jar: https://send.monobank.ua/jar/26mrQPQ3PZ
#donations #Ukraine
A Reddit thread with some useful tools for Kubernetes and kubectl plugins0.
Some things there are well-known, but you may find some new interesting things there. I did :)
#kubernetes
An explainer for the Backend-for-Frontend pattern. The article provides some high-level overview of what it is, and when to use it.
#architecture #design
How Do You Fit a Trillion-Parameter Model Into a Kubernetes Cluster? is an interesting article about how one should change their perspective when reasoning about running LLMs in Kubernetes compared to usual web apps.
It’s an interesting read, especially, if you don’t work with this stuff every day. The biggest takeaway here is that in the case of models, a “replica” doesn’t mean a pod in most of the cases, it’s a distributed system on its own that should behave as one. This article also explains, how exactly things are distributed within a replica, and what are the low level system parameters to pay attention to.
#kubernetes #ai #llm
Unless you're super diligent with deprecation, you may be in a situation right now, when you need to migrate away from NGINX ingress.
Here's a great article that explains new Kubernetes API objects related to the GatewayAPI project that is here to replace Ingress.
Ingress API is not deprecated itself, but it won't be further developed either.
This article confuses the names for the community-led Ingress Nginx and the F5 NGINX ingress controller, but so do many of us: there are way too many nginx's in this world.
#kubernetes #networking #nginx
I Don’t Care if AI Wrote the Code. You Own It. is a reminder that you cannot call AI an idiot, if something goes wrong - you still bear the responsibility of what it does.
This short article just reiterates this statement, and points out that in this day an age, tests and validations are more important as ever before.
#ai #sre
For today’s Donations Monday, let’s finally close the fundraiser for two trucks that’s been going on for some time already.
https://send.monobank.ua/jar/3U1hBa5WPp
More info: https://www.instagram.com/p/DXpgaaWgH00
#donations #Ukraine
A nice article about chaos engineering that was shared in our chat.
The author uses some overly fluffy sentences, but the core of the article holds strong: in many cases, you don't need chaos engineering, and there are things that have better ROI, unless you have them already.
Personally, I'd also like to add that chaos engineering is not simply about breaking things - it's about experimentation. You don't just randomly switch off things, you build hypotheses and validate them. This is the boring, yet crucial part, that many oversee.
#chaos
An interesting point of view on reliability through the prism of everyday work and experience from other industries.
The normal work of creating reliability is an article by Lorin Hochstein, that asks: what instead of thinking of how an incident could have been prevented, we ask: what do we do daily to avoid having incidents constantly.
P.S. "Invert, always invert" - Carl Jacobi
#sre #reliability #culture
A (now) regular Thursday security advisory rubric.
"Fragnesia" is a newly discovered local privilege escalation kernel CVE from the same family of CopyFail and DirtyFrag.
It looks like the Dirty Frag mitigation (disabling the kernel modules esp4, esp6, and rxrpc) should help here as well.
#security
Enabling horizontal autoscaling with co-operative distributed rate limiting is an old article from Monzo that describes, how they built their internal distributed rate limiting solution.
The interesting part is the reasoning about whether a system works in an adversary environment (public facing) or not (internal system). The main question here: can you trust a client? The answer to this question influences the design a lot!
#system_design
A new issue of the CatOps Digest is here!
https://newsletter.catops.dev/p/catops-digest-2026-05-09
Happy Europe Day! 🇪🇺
#digest #newsletter
You may have heard already that Mitchell Hashimoto plans to move Ghostty away from GitHub.
It could be that you plan such a move yourself for whatever reason, but you're not sure yet. Here's a guide on how to push changes to GitHub and Codeberg simultaneously, so you could still keep the door open.
Codeberg is a non-profit European Git hosting. Although, this guide should work for any provider as long as you can use SSH keys for auth.
#programming #github
Continuing with our AI week.
AI in SRE: What's Actually Coming in 2026 is telling a story of AI coming for help with incident response.
The article suggests trying an AI tool for real investigation or data collection for postmortems. To clarify this, in my experience, you don’t need to have a dedicated tool, a general purpose AI agent with some harness (skills and scripts) would do. You should try it! AI does the job of data gathering incredibly well. Yet, the results are indeed not perfect.
Another good point in this article is data quality. AI results are as good as context you provide. I witnessed two prominent failure modes so far:
1. Inference on incomplete data: a person with limited access (typically a developer) asks their agent to investigate an alert. The agent comes to some conclusion. At the same time, a person with elevated access (typically a systems engineer) asks their agent to investigate the same alert and gets a different result, likely because some data is only available via kubectl events, etc. The fix for that is not to allow everyone to do everything, the fix is to revisit your observability pipelines and ensure that you ship all the relevant data, which is easier said than done.
2. Agent that cries "wolves": if you have a pollutant in your logs, or simply an event that happens very often, agents like to correlate it with everything. If your clusters are elastic, an agent could blame node count fluctuations for every error. The problem here is that once node count fluctuation actually causes a problem, you will be the one to ignore this hint from an agent, because it suggests it every single time.
If you are ready to share more AI failure modes specifically related to SRE in Ukrainian, welcome to our chat.
#ai #sre
I will post AI-related articles this week, because why not?
The first one is from Charity Majors called AI demands more engineering discipline. Not less, in which she follows up on her another article.
This one is on technical aspects of moving to the disposable code. It also has a lot of links to other articles, which is also cool.
#ai
A new issue of the CatOps Digest is here!
https://newsletter.catops.dev/p/catops-digest-2026-06-13
#digest #newsletter
How much do amd64 microarchitecture levels help in Go? is a benchmarking article that shows the compute time improvements you can get if you'd build your apps for modern x64 processors only. You likely use modern processors already and do not plan to run your apps on the decade old hardware.
Still, it's important to remember that while such articles are nice; your real applications probably don't just calculate bit vectors all day. It's much more likely your real performance bottleneck is I/O and not the fact that your apps are built with the support for old hardware. Still, you can get some easy wins here by just adding a compilation flag, if you're using Go.
#performance #go #programming
For today's Donations Monday, I'd like to share with you requisites of a friend of mine, who volunteers for AFU since the beginning of the full-scale invasion.
Here's pavlobondarenkopaypal/support-ukrainian-army-with-and-da077ccda388">a page with all the possible ways to donate. You can also find links to the current goals, and reports for previous fundraises there.
Here's their Monobank jar that supports Apple Pay, if you'd better have a direct link:
https://send.monobank.ua/jar/BQjWbpver
#donations #Ukraine
A case study from Amazon, how science solves actual engineering problems that later translate in money savings (likely millions on the Amazon scale).
How a Slack shout-out, a dusted-off academic theory, and a spaghetti monster led an AWS team to crack an elusive code—and deliver greater reliability and performance for customers is a story about AWS realigning their network around the random graph theory.
P.S. I always feel excited about the networking stories, because I studied them in the university. Even though I haven’t worked closely with them since many years ago, and forgot almost everything about them.
#aws #networking
A new issue of the CatOps Digest is here!
https://newsletter.catops.dev/p/catops-digest-2026-05-29
#newsletter #digest
Finding zombies in our systems: A real-world story of CPU bottlenecks is an interesting debugging story for those, who like technical detective tales.
P.S. I find Pinterest's technical blog quite interesting. It has many interesting articles out there.
#debug #aws #performance #kubernetes
Today CatOps became 9 years old 🎉
You can send us a birthday present by donating to our current fundraiser!
https://send.monobank.ua/jar/3U1hBa5WPp
What’s the difference between picking up litter after yourself and donating to the AFU's pickup trucks?
You're right — donating is easier, as you don't need to spend energy producing waste beforehand!
So, here's the link: https://send.monobank.ua/jar/3U1hBa5WPp
More info: https://www.instagram.com/p/DXpgaaWgH00
#donations #Ukraine
Continuing with security advisory.
NGINX ngx_http_rewrite_module vulnerability CVE-2026-42945.
~
NGINX Plus and NGINX Open Source have a vulnerability in the *ngx_http_rewrite_module* module. This vulnerability exists when the *rewrite* directive is followed by a *rewrite*, *if*, or *set* directive and an unnamed Perl-Compatible Regular Expression (PCRE) capture (for example, $1, $2) with a replacement string that includes a question mark (?). An unauthenticated attacker along with conditions beyond its control can exploit this vulnerability by sending crafted HTTP requests. This may cause a heap buffer overflow in the NGINX worker process leading to a restart. Additionally, for systems with Address Space Layout Randomization (ASLR ) disabled, code execution is possible. (CVE-2026-42945)
ngx_http_rewrite_module (and it's widely used!), you are likely vulnerable.
Consistent Hashing in 1 diagram and 198 words is a nice primer on the consistent hashing technique. Obviously, it doesn't go deep on the implementation or examples.
That Substack has some other nice primers as well. Some are good, others are not so much, but all of them could be a good start for a new topic.
#programming #primer
For today’s Donations Monday, I would like to remind you about one of the smaller fundraisers from the recent digest.
- Radio-electronic equipment for the 25th Brigade.
It’s more than 80% complete, and I’m sure that with your help, we can close it this week.
#donations #Ukraine
Remember copy.fail which we all checking a week ago?
Here is a continuation - another Linux 0-day to root.
https://github.com/V4bel/dirtyfrag
Btw, I can recommend to checkout /channel/setenforce_1 - channel fully dedicated to security, or better say - to vulns that will have real effect on you. No bullshit about "10 common vulns" which you can check on OWASP etc. Love it.
#security #linux
A book bundle on Linux, shells, and other OPS topics by O’Reilly on Humble Bundle.
Just beware that this is a reoccurring bundle. It was featured before, including on this channel. Double-check before you buy!
#books