DevOps and other issues by Yurii Rochniak (@grem1in) - SRE @ Preply && Maksym Vlasov (@MaxymVlasov) - Engineer @ Star. Opinions on our own. We do not post ads including event announcements. Please, do not bother us with such requests!
From one conference to another!
On 22nd of February (which is sooner than it may appear) I’m speaking at the DevOps fwdays’25 about the pitfalls of a platform team and caveats you can encounter when building one. The talk is online and in English, so you can join from anywhere you want!
Also, here’s a promo code for 15% off from your ticket: 4A91C4612C
#event
Ok. I think I can start sharing things from FOSDEM / CfgMgmt Camp conferences.
To my knowledge, the videos are not available yet, but here are some pretty self-explanatory slides from the opening talk of the PostgreSQL panel: PostgreSQL Performance - 20 years of improvements.
In this talk, the author - a core contributor of PostgreSQL - presets the benchmark results of 10 major versions of PostgreSQL, so you can see how it evolved over the years.
A link to the talk page
#databases #postgres #fosdem
The first newsletter issue of 2025 is here!
https://newsletter.catops.dev/p/catops-digest-2025-02-02
It should have came earlier, but we live in the real world.
#digest #newsletter
If you missed O’Reilly’s “Linux for seasoned admins” books bundle, you can get the same books in a new bundle that is now called “Linux from beginner to professional”.
There are some good books in that bundle, BTW.
#books
There are many different release patterns. Today I'd like to share with you one that is called "dark reads". This one is hard to google, because you get a lot of articles either about dark patterns of web development or a bunch of photos of dark red wallpapers.
Here's one article I could find about it. The core idea is that you have both versions v1 and v2 running at the same time, but you only read and outputs from v2 and check that for correctness.
It is very similar to canary releases or feature flags, but it's more applicable to data pipelines. If you have a stream of messages, it's not always easy to stream a percentage of them via a new system. Also, when running two versions at the same time, you need to have additional logic to deduplicate messages at some point. "Dark reads" says that you should be able to validate and discard v2 events on the consumer side and flip the switch once you're satisfied with the results.
#design
How AI-assisted coding will change software engineering: hard truths is a sort of "wrapped article". You can call it a "reaction article" or an "enriched article" if you with.
It has two parts: the original post about AI usage in development teams by Addy Osmani, I think I already shared this article on this channel.
The second part are comments by Gergely Orosz - the author of Pragmatic Engineer - on the topic.
I think, those comments are valuable and do provide additional input and perspective instead of just parasitizing on existing content, as many reaction videos do.
So, enjoy! Especially, if you haven't read the first article.
#ai
I know that oftentimes folks are critical towards Packt, because many books and courses there are not that good. However, some of them could still provide value.
Thus, it's nice to get them at a discount. Right now, Humble Bundle has two discounts on Packt products:
- Books bundle about databases
- Courses for AWS certification
I might actually get the one on databases, because I want to learn more about Postgresql, but I don't have any specific resource in mind except reading the docs.
Speaking of certifications, we had a voice chat recently on that topic. The voice chat is in Ukrainian.
#books #databases #aws
It seems like this week is a week of cultural posts, so be it.
This one is as cultured as it gets, since it explores a technic of Socratic questioning towards the idea that meetings are the waste of time.
You can try it in your team, if you like. Maybe, you'll be able to remove some meetings or re-arrange them in a better way. Or maybe, you really need more meetings, not less - who knows.
#culture
An article on how to foster a culture of ownership in your company. Ownership culture is crucial for engineering success.
Yet, I have a problem with such articles: they are always written from the perspective of a senior manager or a high-level individual contributor. In this case, the author describes the time he was a Principal Engineer.
I am yet to see such an article from an individual contributor's perspective. At this point, I am not even sure, if it's possible to make grassroot changes in your organization, given that the organization is big enough. If you have a success story, please, let me know!
P.S. This particular Substack also have a clutter problem: there is a sponsor part in the beginning and a self-promotion part in the end, which you can skip.
#culture
Observability in the Age of AI is an article by Charity Majors and Phillip Carter that speculates about the changes that observability stacks and engineering teams alike will face with an introduction of AI.
This is an article from the company, so of course you have self-promotion here, but there are also very good points:
- If you can compute the answer, you probably should compute the answer
- Often people use AI to try to close the gaps of data they do not collect, yet AI results would be also better if that data is present
- When talking about people, AI is better used as an additional reviewer rather than a creator
And one thing that is not directly related to observability, but has given me a lot of food for thoughts:
- We expect software will bifurcate into two categories: disposable code and the code that runs the world.
I've put direct quotes in italic.
#observability #ai
An interesting article by Google about the evolution of SRE.
In nutshell, they describe how they can apply the control theory principles towards software systems to see how can not just individual components fail, but also what are the failure patterns of those components working together.
#sre
For today’s Donations Monday, I want to share with you a fundraiser from one of our community members.
This is a fundraiser for a mini-excavator for the 36th separate brigade of AFU, which will tremendously help them to dig trenches in winter.
https://send.monobank.ua/jar/3aq2sBa6Fk
#donations #Ukraine
An online calculator for DataDog costs + some tips on how to keep these costs in check.
Yes, this is a marketing material, so you need to take the "comparison" part with a kilogram of salt. Also, the calculator itself is tricky: DataDog has many ways of "committing" to some capacity, which impacts the price drastically. They mention this fact in the article as well. So, this calculator is rather a reference than a precise tool.
Otherwise, this article provides some sane tips on the DataDog cost management.
#costs #finops #observability
Asking questions during an interview is as important as answering them, one could argue.
Here are a couple of articles with examples of what to ask your potential employer and the team.
- Reverse Interviewing Your Future Manager and Team - an old article from Gergely Orosz on the topic. Kudos to him for not closing old articles behind a paywall as many Substack creators do!
- 40+ questions to ask at a job interview - the website itself is Berlin-centric, but these questions are independent from a location.
Also, one of the top motivations to search for a new job is compensation. So, if you have spare 10 minutes this Sunday afternoon, folks from DOU would greatly appreciate if, if you could participate in their annual salary survey (in Ukrainian). This year they collect all the answers, including people who do not live in Ukraine (hopefully, those answers won’t be filtered out in the end results as well).
#jobs #salary #survey
Another thing from FOSDEM PostgreSQL devroom is pgroll - a tool that allows one to execute zero-downtime (zero lock-time really) migrations in PostgreSQL.
Slides are available at the talk page. I hope, the video will be available soon as well. It was a very good talk!
#databases #postgres #fosdem
There’s a new fundraiser by Come Back Alive has a new fundraiser for communication equipment for the army aviation.
As someone who studied telecommunications back in a day, I have a soft spot in my heart for everything related, even though I don’t remember shit about telecommunications.
P.S. Unfortunately, smaller fundraisers do not get much traction any more. Hopefully, a bigger fundraiser will go better.
#donations #Monday
A small list from HoneyComb with good practices to organize your logs.
It's in a form of a numbered list (the peak Internet format, he-he), so it is easy to read. If you're a seasoned engineer, there won't be any unexpected points for you. However, from my experience, log management is often left as an afterthought. So, maybe, this article could serve as a neat checklist for you.
#observability
We continue with the fundraiser for FPV drone details for the KORD unit. The fundraiser is done by my wife's friends.
It's still about 20k UAH to go!
https://send.monobank.ua/jar/4WLw91UqFe
#donations #Ukraine
One of the strengths of Go is the existence of default helper tools such as gofmt
and goimport
. Thus, you don't have to worry about formatting, for example. These tools can also be easily integrated into your editor of choice.
For quite some time now, I have to write Python code in professional setting and an automated import is one of the things I miss the most. There are tools to achieve that, ofc, but still.
Another thing about such helper tools is that we often take them for granted, especially when they are default. But you don't have to!
This article describes on a high level how goimport
works. While it is kinda intuitive, it's always nice to see a bit deeper perspective. And if you want to dive even deeper, you can always just read the code yourself.
#go #programming
For today’s Donations Monday, I’d like to remind you about this fundraiser for parts for FOV drones for the KORD unit.
https://send.monobank.ua/jar/4WLw91UqFe
BTW, if every subscriber donates just €0.15 towards this goal, this fundraiser will be closed.
#donations #Monday
It is this time of the year: many companies are doing their performance reviews and calibrations. So, make sure to recall what you did last year, write something nice about your peers. Keep calm and carry on.
Here is a nice small write-up on performance reviews, specifically its dark side. One thing to take away from there:
> If you want them to MAYBE change, talk to them directly. If you want them to get stabbed by management, put it in their performance review.
Here's also an opinion piece from Rory Sutherland - a vice chairman of the Ogilvy & Mather group - where he explains how individual performance goals jeopardizes risk taking with potential sharable results.
P.S. Another interesting topics when talking about performance reviews in the way they are today is the use of AI. I know for sure that people use it to write those reviews, and it can be definitely used to analyze a huge influx of reviews. Thus, we are one step closer to a singularity in which machines will write reports for other machines to read, so us, flesh bags, can focus on our work.
#culture #bureaucracy
For today's Donations Monday, I would like to remind you about a fundraiser to support a friend of my wife, who is currently serving with the KORD (Special Purpose Police Unit) on the front lines
in Donetsk.
His team is in urgent need of essential components for
their FPV (First-Person View) drones, which are critical for
surveillance and reconnaissance in the region.
The funds will go towards purchasing:
- Repeaters for better signal range
- Antennas for stable communication
- Batteries for extended flight time
Your support will help KORD improve their operations and stay safe while protecting Ukraine.
https://send.monobank.ua/jar/4WLw91UqFe
Every donation matters!
#donations #Monday
Seva from UkrOps has started an English version of his blog with the first post about handling concurrency in Bash!
Bash still does a lot of heavy-lifting inside automation tools, task managers, and "CI/CD" systems, as well as for automating mundane ad-hoc tasks. So, you can easily apply ideas from this article in your setup.
#bash #programming
It’s time to get out from the holidays hibernation and get back to this channel again. And what could be the best way to start a year, if not with a donation for AFU?
Wife of my friend from my very first tech-related job raises 15k UAH for AFU. There’s also a raffle to win a certificate to buy books.
The goal is not that high, so I’m pretty sure we can close it today together!
Monobank Jar:
https://send.monobank.ua/jar/72CpNXHERf
#donations #Ukraine
An almost New Year episode of our voice chat (in Ukrainian).
At the end of this year, we gathered to discuss certifications: who needs them and why, and how to obtain them. You can find the episode on:
- YouTube
- Substack
- Spotify
- Apple Podcasts
Enjoy this new episode while making some New Year salads!
P.S. Please, let me know in the YouTube comments or elsewhere, if you like the voice chat format and if it makes sense to continue it in the new year. Cheers!
#voice_chat #certifications
Breaking down OpenAI's outage: How to avoid a hidden DNS dependency in Kubernetes - in this article, folks at Render analyze the recent OpenAI outage and provide some insights on the possible cause of the issue.
This article also serves as a reminder that you'd better run DNS and the Control Plane on separate nodes and also that the data in ETCD is organized by namespace, thus listing "all" resources is very costly, especially in large clusters, since you need to scan everything.
#kubernetes #postmortem
For today's Donations Monday, we’re raising funds to support a friend of my wife, who is currently serving with the KORD Special Purpose Police Unit on the front lines in Donetsk.
His team is in urgent need of essential components for
the FPV drones.
The funds are for:
- Repeaters for better signal range
- Antennas for stable communication
- Batteries for extended flight time
Your support will help KORD improve their operations and stay safe while protecting Ukraine.
https://send.monobank.ua/jar/4WLw91UqFe
Every donation matters!
#donations #Monday
I didn't pay enough attention yesterday, so I didn't post the second part of the article about Argo CD :\
Let's fix it! Here it is. This second part focuses on the .source
configuration parameter of ApplicationSet
.
#cicd #argocd