5629
DevOps and other issues by Yurii Rochniak (@grem1in) - SRE @ Preply && Maksym Vlasov (@MaxymVlasov) - Engineer @ Star. Opinions on our own. We do not post ads including event announcements. Please, do not bother us with such requests!
An article from Anthropic on the AI agents evaluation strategies, basically test strategies for AI agents.
It's an interesting read, even if you do not create your own agents. There are also some interesting points in the difference of evaluating a coding agent vs a communication agent (customer support, etc.), as well as examples of the metrics you can use for that.
Besides, you can use the ideas from this article not only for the agents, but also for skills, if you develop any.
#ai #testing
An interesting article by Coroot on reproducing split brain on CloudNativePG.
Regardless of whether or not you use CloudNativePG, it's a good example of chaos testing. Chaos testing is about an experiment: here they define a clear gap in certainty (what would happen if?) and tested it.
The article also has some recommendations on CloudNativePG configuration, if you're using it.
#databases #postgres #kubernetes
You likely already know about the recent security incident that involves OpenAI and Hugging Face, when a model escaped its containment to cheat an evaluation test and hacked Hugging Face in the process. Now, postmortems are available from both OpenAI and Hugging Face.
- Initial incident disclosure by Hugging Face
- Postmortem by Hugging Face
- Postmortem by OpenAI
This is quite interesting read. On one hand, in the words of Hugging Face: "AI-driven offensive tooling is no longer theoretical. It lowers the cost of running a broad, patient, multi-stage campaign, and it operates at machine speed. Defending an online platform now means treating the data and model surface as a first-class attack surface, and using AI on defense to keep pace."
But also: "...we had no admission policy rejecting privileged or hostPath pods, and the CSI driver's ClusterRole granted pod creation cluster-wide..."
This highlights not only the need to develop new defense methods and strategies, but also the fact that the good-old multi-layer security approach is important, and old boring things work as well.
#ai #security
We continue with a fundraiser a pickup truck for the 423rd battalion.
Right now, one jar has 218/350k raised, and the other one 37/50k.
Jar 1:
https://send.monobank.ua/jar/2aMqtZT592
Jar 2:
https://send.monobank.ua/jar/8oTyqJUjPV
#donations #Ukraine
Plantescale (a database company) released an article about sharding.
It doesn't go into deep technical details. Also, it's a pity that they did not touch on some operational aspects of sharded storages: how to back up those systems, replication, etc. On another hand, because of this the article is brief and easy to follow.
It may be useful for your system design interviews out there, because it is high level and it mentions some existing out-of-the-box solutions.
P.S. It also has some very nice animations!
#databases
We continue with a fundraiser a pickup truck for the 423rd battalion.
Right now, one jar has 218/350k raised, and the other one 32/50k.
Jar 1:
https://send.monobank.ua/jar/2aMqtZT592
Jar 2:
https://send.monobank.ua/jar/8oTyqJUjPV
#donations #Ukraine
Rewriting everything in Rust was a meme of itself before the wide application of AI. However, Rewriting Bun in Rust is interesting precisely because of AI.
Quoting the article itself:
Pre-merge, this took 5.9 billion uncached input tokens, 690 million output tokens, and 72 billion cached input token reads — around $165,000 at API pricing. By hand, I think this would've taken 3 engineers with full context on the codebase about a year, during which time we wouldn't be able to improve Node.js compatibility, fix bugs, fix security issues or implement new features. We never would've done that. The realistic alternative was to do nothing and keep fixing the bugs at the top of this post forever.
What’s interesting in this article, regardless of whether or not you plan to rewrite a huge code base, is how he approached the task with multiple agents. This is something you could do in your daily job for smaller things as well.
Bun v1.4.0 is the first Rust version, and some companies already use it in production.
#ai #programming #rust
My friend raises money for a pickup truck for the 423rd battalion.
Right now, there are 200 out of 350 k UAH raised.
You can donate on a supportive jar:
https://send.monobank.ua/jar/2aMqtZT592
Or on her jar:
https://send.monobank.ua/jar/8oTyqJUjPV
#donations #Ukraine
Some book bundles on Humble Bundle:
- AI usage and practices
- Linux things
Just remember to always check, if you have the books already, because these bundles repeat from time to time.
#books
A post from Cloudflare about a low-level race condition they tracked down and fixed in the Rust Hyper library.
I like reading such detective stories. Also, I recall times, when people would regularly ask about strace on the interviews. I am not sure if this is still the case. At least, I wasn’t asked about strace for a long time.
#programming #postmortem
The four horsemen behind thousands of Postgres outages is a self-promotion article, but it can teach you some things about Postgres, so I allow it.
A few corrections, though. Postgres does have a pg_hint_plan extension that allows you to modify the plan yourself. However, if you need to use that, there may be something odd with your queries in the first place.
The second thing is JSON. Postgres works with it and a lot of people use JSON fields, but this database was not created for JSON in the first place. So, if you need to work mostly with JSON, you would probably be better with another storage, or you could deserialize JSON fields into columns and work with data as usual.
#databases
A big fundraiser from DOU for the 2nd separate corps of the National Guard of Ukraine «Хартія» is still ongoing.
The goal of this fundraiser is to buy heavy bomber drones "Vampire" for the Kupiansk direction.
Monobank jar: https://send.monobank.ua/jar/26mrQPQ3PZ
#donations #Ukraine
An article about optimizing the symbolicator - the part of the observability stack that translates stack traces of minified code into human-readable ones.
It’s an interesting read about what a design optimization can achieve. In the discussion on Reddit, commentators rightfully pointed out that the drastic difference between this new symbolicator and the baseline is due to the approach that author uses, and that a C/Rust version would still perform batter compared to the example in Go. Yet, this is kinda the point: by designing your application in a clever way, you can achieve better performance with “slower” technologies compared to brute-forcing the solution using “faster” technologies.
#programming
We continue supporting DOU with their fundraiser for the 2nd separate corps of the National Guard of Ukraine «Хартія». The goal of this fundraiser is to buy heavy bomber drones "Vampire" for the Kupiansk direction.
Monobank jar: https://send.monobank.ua/jar/26mrQPQ3PZ
#donations #Ukraine
Continuing with our AI week.
AI in SRE: What's Actually Coming in 2026 is telling a story of AI coming for help with incident response.
The article suggests trying an AI tool for real investigation or data collection for postmortems. To clarify this, in my experience, you don’t need to have a dedicated tool, a general purpose AI agent with some harness (skills and scripts) would do. You should try it! AI does the job of data gathering incredibly well. Yet, the results are indeed not perfect.
Another good point in this article is data quality. AI results are as good as context you provide. I witnessed two prominent failure modes so far:
1. Inference on incomplete data: a person with limited access (typically a developer) asks their agent to investigate an alert. The agent comes to some conclusion. At the same time, a person with elevated access (typically a systems engineer) asks their agent to investigate the same alert and gets a different result, likely because some data is only available via kubectl events, etc. The fix for that is not to allow everyone to do everything, the fix is to revisit your observability pipelines and ensure that you ship all the relevant data, which is easier said than done.
2. Agent that cries "wolves": if you have a pollutant in your logs, or simply an event that happens very often, agents like to correlate it with everything. If your clusters are elastic, an agent could blame node count fluctuations for every error. The problem here is that once node count fluctuation actually causes a problem, you will be the one to ignore this hint from an agent, because it suggests it every single time.
If you are ready to share more AI failure modes specifically related to SRE in Ukrainian, welcome to our chat.
#ai #sre
For today's Donations Monday, I'd like to remind you about a big fundraiser by DOU for the 2nd separate corps of the National Guard of Ukraine «Хартія».
The goal of this fundraiser is to buy heavy bomber drones "Vampire" for the Kupiansk direction.
Monobank jar: https://send.monobank.ua/jar/26mrQPQ3PZ
#donations #Ukraine
A book bundle on Python, Rust, and Data by O’Reilly on Humble Bundle.
https://www.humblebundle.com/books/python-rust-and-data-oreilly-books
The bundle is active for 18 more days. Enjoy!
#books
Being a good communicator is becoming more and more important in the world, where more and more code is generated.
15 principles for managing up is an article with some tips that can help you not only get on a better foot with your manager, but also to become a better colleague in general.
I personally so not like the term “managing up”, but I guess you have to call it somehow.
#culture #mgmt
Another book bundle on HumbleBundle. This one is about software architecture. Hopefully, you’d find something interesting there.
https://www.humblebundle.com/books/software-architecture-apress-books
#books
From all the management blogs, stories from Mike Fisher always resonate with me somehow. At least, those stories that get to me.
You Can’t Fake Belonging is a story of that feeling that is hard to put into words, but that greatly defines if your job is an interesting journey or a soul crushing routine.
From the first few paragraphs, I assumed that this would be yet another "office good" article, but it isn't! Let him cook and hopefully, this article would be insightful for you as well!
What I especially liked about it is the description of the manager's role. I guess everyone can name a good and a bad manager they had in the past, but it's not always clear why one of them was good and another one - bad. Also, the last bit hits hard: "The most dangerous person on your team isn’t the loudest critic or the lowest performer. It’s the one who has quietly decided they don’t belong here, started to believe it, and hasn’t left yet."
#culture #mgmt
I often post book bundles on this channel, but buying books and reading books are two different hobbies.
Here’s a book list on Github with software engineering books selected by a set of criteria:
- The book must be relevant to Software Architecture; and
- The content is not obsolete.
- It must be technology agnostic or applicable to a wide range of technologies;
- An average rating of 3.5 or above; and
- A total number of ratings equals 50 or above.
- At least 5 written reviews.
- A minimum length of 50 pages.
Hopefully, it could help you to find what to actually read.
#books
Multi-Agent System Reliability is an article by Alex Ewerlöf that touches on some practical aspects of working with agentic AI systems day-to-day.
It covers patterns you can use to improve the output, and probably the main point:
> stop treating LLMs like magic chatbots. Start treating them like unreliable components in a distributed system.
#ai #sre
A new issue of the CatOps Digest is here!
https://newsletter.catops.dev/p/catops-digest-2026-07-11
#digest #newsletter
Yesterday Flux turned 10 years old! 🎉
In this article they reflect on this journey and recall some pivotal moments from the past. Plus, highlight what are they doing now.
P.S. Do not forget to update your CV 😁
#kubernetes #gitops #flux
Finally, easy AWS EKS rollbacks to previous K8s version!
Now you can trust EKS upgrade even to your AI agent (please don't)
https://aws.amazon.com/blogs/aws/upgrade-amazon-eks-clusters-with-confidence-using-kubernetes-version-rollbacks/
#kubernetes #eks #aws
Term “gateway” is super-widespread in the Kubernetes world. One of the recent additions is, of course, the AI Gateway. But what is that? Obviously, a reverse proxy, but what else?
This article aims to answer this question. The most important part here is that it doesn’t try to tell you what gateway is the best, but rather outlines subtle differences between the flavors. So, you could choose responsibly, if you need such a gateway.
#ai #kubernetes
Save a list of Kubernetes defaults in one place, so you don't forget, and don't need to find them every time.
#kubernetes
Not all index scans are equal is an article by Datadog, where they describe the idea of targeted DB indices and when to use those.
There is also some praise for their database monitoring tooling, but this is a vendor article after all.
The only thing is that they didn't mention that too many indices also comes with a price: you need to store and update them. So, always evaluate the performance for some period of time after adding indices.
#databases #observability
So, that's for AI in the companies, but what about AI in the wild i.e. in open source?
We have cases like curl, that had to take down their bug bounty program due to the influx of slop bug reports. Yet, the industry adapts.
Here's a study by Redmonk on the stance of various foundations and standalone open source projects on AI, including their major concerns, and openness to AI-generated contributions.
#ai #open_source
Harness engineering for coding agent users is a new guest article in Martin Fowler's blog that summarizes approaches to improve AI output and make it more manageable.
If you're actively using AI agents day-to-day, things described in this article won't be news to you, but it helps to structure one's thoughts.
#ai