CatOps

25 November 2025 17:33

Do you have the "What went well" section in your postmortems?

Here's an argument to have one with explanation of why this is important.

tl;dr: Because while each incident is different, there is a set of skills and behaviors that allow one to improvise under pressure to mitigate an incident. These skills and behaviors can be taught as well, and your "What went well" section is also for that.

#sre #incidents

CatOps

22 November 2025 13:11

Always Be Ready to Leave (Even If You Never Do) is not about keeping your CV up-to-date or socializing with recruiters, as it may seem from the title. It’s a short article on work habits that would keep you more efficient and, probably, happy at work; even if these habits would eventually make it easier for you to quit, if you choose to.

#culture

CatOps

19 November 2025 08:54

A postmortem from Cloudflare for yesterday’s outage is now available.

tl;dr:
>>>
The issue was not caused, directly or indirectly, by a cyber attack or malicious activity of any kind.Instead, it was triggered by a change to one of our database systems' permissions which caused the database to output multiple entries into a “feature file” used by our Bot Management system. That feature file, in turn, doubled in size. The larger-than-expected feature file was then propagated to all the machines that make up our network.
<<<

Another interesting thing:
>>>
Unrelated to this incident, we were and are currently migrating our customer traffic to a new version of our proxy service, internally known as FL2. Both versions were affected by the issue, although the impact observed was different.
Customers deployed on the new FL2 proxy engine, observed HTTP 5xx errors. Customers on our old proxy engine, known as FL, did not see errors, but bot scores were not generated correctly, resulting in all traffic receiving a bot score of zero. Customers that had rules deployed to block bots would have seen large numbers of false positives. Customers who were not using our bot score in their rules did not see any impact.
<<<

So, if you were not affected yesterday, you know why now.

#postmortem #cloudflare

CatOps

17 November 2025 16:01

​​For today's Donations Monday, I'd like to ask you to donate to the administrative needs of the "Come Back Alive" foundation.

It takes tremendous effort to run a foundation like this, and despite they can, they do not take money for the operational needs from regular donations. Thus, it's important to help them cover those needs as well!

https://savelife.in.ua/en/donate-en/#donate-fund-card-once

#donations #Ukraine

CatOps

13 November 2025 08:00

Press F to pay respects.

>>> Ingress NGINX Retirement: Kubernetes SIG Network and the Security Response Committee are announcing the upcoming retirement of Ingress NGINX. Best-effort maintenance will continue until March 2026. Afterward, there will be no further releases, no bugfixes, and no updates to resolve any security vulnerabilities that may be discovered. Existing deployments of Ingress NGINX will continue to function and installation artifacts will remain available.

Announcement page.

#kubernetes #nginx

CatOps

11 November 2025 09:36

An interesting lab for an overengineered solution from AWS for Kubernetes workloads right sizing.

Should you implement it this way? I don't know. But maybe, you want to play with GitOps, AWS Bedrock and all that stuff.

Also, it's funny how they say in the beginning that having VPA and Goldilocks inside a cluster is an overhead and additional management burden and then propose to create a cluster in GHA runtime and use generative AI to address that.

#aws #kubernetes

CatOps

09 November 2025 15:59

An article by Charity Majors on why thinking of Observability in pillars is limiting.

I recall a similar article from the past about how Facebook does their observability. It’s somewhere here on the channel.

The core idea is to treat all the signals as universal wide events that would allow one to preserve all the context and not hop between different tools.

#observability

CatOps

05 November 2025 08:30

More follow-ups for the AWS outage (Azure outage didn't generate that much press).

Lorin Hochstein analyzes the postmortem from the complexity point of view and comes to quite interesting conclusions that you can absolutely apply to your incidents and postmortems as well.

tl;dr is that incidents (especially bigger ones) are often unique. So, when reasoning about the preventive measures, you need not only to prevent similar incidents, but also get prepared to handle incidents in general, because the next incident may be not the same as the present one.

#reliability #sre #aws

CatOps

31 October 2025 17:45

A new issue of the CatOps Digest:

https://newsletter.catops.dev/p/catops-digest-2025-10-31

#digest #newsletter

CatOps

29 October 2025 15:30

Reddit engineers made a long post about their DevEx survey, which they shared... on Reddit.

This is a nice read, if you'd like to learn how different companies evaluate developer productivity and satisfaction.

A few interesting points there:

- They don't use any specialized SaaS tools for this survey - only Typeform and Looker.
- The survey is quite long, but the adoption seems to be good.
- They include various topics into the survey, not just purely DX metrics, they also changed the name of the survey to reflect that.
- Teams can use the results on their own to calibrate their decisions.

#dx #reddit

CatOps

22 October 2025 14:05

I don't know if you are involved into the promotions cycles in your company or participate in hiring, but I still want to share with you this article - On Hiring: Promote Stars, Not Strangers.

It is targeted towards managers, but you can easily apply the core idea of this article to any role or position in a company.

#culture

CatOps

20 October 2025 11:12

​​Speaking of AWS:

Oct 20 2:01 AM PDT We have identified a potential root cause for error rates for the DynamoDB APIs in the US-EAST-1 Region. Based on our investigation, the issue appears to be related to DNS resolution of the DynamoDB API endpoint in US-EAST-1. We are working on multiple parallel paths to accelerate recovery. This issue also affects other AWS Services in the US-EAST-1 Region. Global services or features that rely on US-EAST-1 endpoints such as IAM updates and DynamoDB Global tables may also be experiencing issues. During this time, customers may be unable to create or update Support Cases. We recommend customers continue to retry any failed requests. We will continue to provide updates as we have more information to share, or by 2:45 AM.

#aws #outage

CatOps

19 October 2025 17:03

Nice weekend read about technical blogging.

https://writethatblog.substack.com/p/thorsten-ball-on-technical-blogging

One of the best ways to learn is to try to explain whatever you have learned to someone else. Blogging is a form of that.

Perhaps, we can dedicate a day to share the blogs we like here on CatOps, let me know, if you think, it’s a good idea.

#culture

CatOps

16 October 2025 17:00

A bundle of Python programming books by Pearson.

While Python won't be my first choice, sometimes it's inevitable. Also, Pearson usually have great books!

#books #programming #humblebundle

CatOps

13 October 2025 11:38

​​It’s Monday, fellas.

A friend of mine is raising money for a Starlink and 3 power stations for the 5th Separate Heavy Mechanized Brigade and 33rd Separate Mechanized Brigade.

You can donate via to this Monobank jar (Apple pay works):

https://send.monobank.ua/jar/3VEQNLAcia

Almost half of the goal is already there.

#donations #Monday

CatOps

24 November 2025 16:19

​​For today’s Donations Monday, I would like to remind you about the foundation that we’ve been partnering with for DevOps Days Ukraine for years now.

UA Responders. Their specialization is medical equipment and such.

#donations #Ukraine

CatOps

20 November 2025 12:09

It's been a while since we had simple how-to articles here. So, here you are:
alipandidan/jenkins-jvm-monitoring-with-remote-jmx-c9477024d23b">How to enable the JMX port on Jenkins.

It's short and actionable, and you would be surprised to learn how many people use Jenkins till these days.

#ci #java #debug

CatOps

18 November 2025 18:31

We don't know why Cloudflare is down - their status page is not so detailed as one of AWS.

However, you can still check out some books on Humble Bundle:

- Data engineering & data science by O'Reilly.
- Software architecture by Pearson.

#books #bundle

CatOps

14 November 2025 14:24

A new issue of the CatOps Digest is here:

https://newsletter.catops.dev/p/catops-digest-2025-11-14

#digest #newsletter

CatOps

12 November 2025 17:27

For people nostalgic for on-premise setups, Dropbox reviled their new generation hardware setup and the challenges they face storing exabytes of data.

#on_prem

CatOps

10 November 2025 13:15

​​For today's Donations Monday, I'd like to share with you a fundraiser for the Optic Dragons unit - a specialized FPV drone assault unit of the 92nd Separate Assault Brigade.

They're raising funds for optical fiber drones, spare parts for converting drones to fiber optics, and supporting combat vehicles of pilot crews. The unit has been redeployed to the Pokrovsk direction where the situation is intense and they need more drone reels for optical drones.

Direct donation link:
https://send.monobank.ua/jar/7D7whfQHfF

Card number: 4441 1111 2291 2961

#donations #Ukraine

CatOps

07 November 2025 19:03

A book bundle that I wanted to post a couple of days ago, but forgot. So, here it is:

Linux for Professionals by Apress.

#books #linux

CatOps

03 November 2025 15:33

​​For today’s Donations Monday, I’d like to share with you a fundraiser for FPV drones from DeepState - a collective behind the close-to-real-time battlefield maps.

https://send.monobank.ua/jar/9AtiB8esqu

#donations #Ukraine

CatOps

30 October 2025 11:50

OneUptime has published their update after two years since moving from AWS.

It's an interesting read, and tl;dr is that they do not regret their decision at all. Although, they admit that they still use cloud for dev environments, cold storage, etc.

Here are a few points that I find interesting:

- Our workload is 24/7 steady / We still recommend staying put if your usage pattern is spiky or seasonal - right sizing is one of the major advantages of clouds that is often overlooked.
- We still recommend staying put if you lean heavily on managed services - this is another important point. Managed services add a lot of value to the clouds. It does seem a bit dumb to use AWS just like an expensive datacenter. On another hand, if you want to be able to do multi-cloud, hybrid-cloud, etc.; you have to make a deliberate decision to stay as much decoupled from cloud offerings as possible. It's a deliberate strategy that trades flexibility for immediate value.
- Ceph stack in production - I'm sure, Ceph evolved a lot through the years, but I still remember words of a colleague of mine from a long time ago: "We didn't lose the data, we just cannot retrieve it". Back then, we decided to keep on-premise installations with ephemeral disks and ship all the data that has to be preserved into AWS (it's not like there was a lot of data to preserve there, though).
- so we added Anycast ingress via BGP with our transit provider to cut traffic shifting to sub-minute and We PXE boot with Tinkerbell - ask on an interview about Anycast, and the same day you will get an angry post on Reddit about unreasonable questions and lowballing candidates, lol. Or maybe, it's just my pessimism speaking.

Anyway, use your best judgment before doing any rapid moves. BTW, this advice is generally applicable, and is not limited to the clouds vs dc discussions.

#aws #bare_metal

CatOps

24 October 2025 10:30

A glimpse into the alerting infrastructure of Hugging Face - a repository for ML models and datasets.

This article has a bit of a "they made me write this for promotion" vibe, but it's still interesting to see what technologies other people use, even if they don't dig deep into any of them.

#observability

CatOps

21 October 2025 11:41

Helm graduated into a "boring technology" a long time ago. So, the news about the upcoming v4 major version are probably no so exciting, unless you are a Helm plugin developer.

Anyway, here's a sneak peek of upcoming changes. They promise that:

>>>
*Bottom Line*: Helm v4.0 is a major architectural upgrade focused on better Kubernetes integration, enhanced plugin capabilities,
and improved developer experience while maintaining chart
compatibility.

So, if you only care about your charts, those should continue to work.

#helm #kubernetes

CatOps

20 October 2025 11:07

​​Since AWS has an outage, some of you have unplanned time off anyway. So, it's a good time to make a donation to a noble cause!

A friend of mine has a supportive jar for INSCIENCE who partnered with the Come Back Alive foundation to raise money to combat enemy UAVs.

https://send.monobank.ua/jar/fKfgmjgw1

Her goal is 20k UAH, so we can easily achieve it!

P.S. Apparently, Monobank hosts in AWS, since their web paged did not renew once I sent the donation.
P.P.S. Despite N26 also being on AWS, they processed the transaction just fine.

#donations #Ukraine

CatOps

17 October 2025 18:25

A new issue of the CatOps Digest is here!
https://newsletter.catops.dev/p/catops-digest-2025-10-17

One week later than it should have been released, but oh well...

#digest #newsletter

CatOps

14 October 2025 17:57

I like it, when people talk about definitions, especially for some common / widely use terms. The more common a term is, the more one is "afraid" to ask about it. And since we cannot glimpse into other people's minds, we can talk about completely different things using the same words.

Availability Models talks about the definitions of "high availability" - incredibly popular term in computer science! It doesn't examine all the availability models, despite its name. Rather, this article brings up a question: so, what the heck is "high availability" and how can we define that based on our actual needs?

P.S. Also, I didn't know about the PACELC theorem. It always feels so cool to learn something new!

#databases #programming #system_design

CatOps

10 October 2025 18:51

There’s a lot of fluff around about how one should be a technical leader, yet the exact understanding of what is leadership is rather vague.

Here’s a nice Friday read on the topic that resonated with me.

Here’s a great quote:

>>>
The more comfortable you become with not being the expert, the more effective you become as a leader.
When you stop trying to out-expert the experts, you can focus on what expert teams actually need:
- Clear problem definitions
- Context for decision-making
- Translation between different perspectives
- Protection from unnecessary complexity
- Space to do their best work

#culture