CatOps

27 Feb 2024 14:22

Resend had a 12 hour outage on the 21st of February.

tl;dr:
> The database migration accidentally deleted data from production servers…
> … we performed a database migration command locally, but it incorrectly pointed to the production environment instead…

You can read it in more details is the article, but here are some of the action items from this postmortem:

- No accessible user role should have write privileges on the production database.
- Improve local development to reduce risks related to database migrations.
- Create redundancy to preserve sending function even during a database outage.
- Increase cadence for disaster recovery tests.
- Implement incident banner on Resend dashboard to inform users quickly.

So, I dunno, check your database. Maybe, you have such a risk as well.

Also, it’s kinda strange that people rarely talk about network isolation not only between their production and non-production environments, but also between their local environment and production. Make production access conscious. Put it on a separate role/VPN. Add some friction accessing it.

Moreover, for the love of god, validate your DB backups.

#postmortem #databases

CatOps

24 Feb 2024 10:13

Today marks the 2 year of the russian full-scale invasion of Ukraine.

Today there are going to be rallies in many cities around the globe.

You can find the list here:

https://ukrainianvictory.org/publications/announcement-pro-ukrainian-rallies-marking-the-second-anniversary-of-the-russian-invasion-updated/

Go there, take your family, take your friends, take your colleagues.

Remind you local politicians that war is not over and that there would be no peace in this world until ruzzia exists!

CatOps

21 Feb 2024 15:55

For you to know: the full Kubernetes CKS (certified security specialist) is available on YouTube. There are both theory and practice, but obviously you will need to take the exam separately.

#kubernetes

CatOps

19 Feb 2024 14:04

A friend of mine is raising money for an FPV-drone Bomber Insomnia for defenders from the 119 separate brigade of Territorial Defence.

You can donate directly to this Monobank Jar: https://send.monobank.ua/jar/LmRuSA8dm

#Donations #Ukraine

CatOps

17 Feb 2024 09:30

​​AI, AI everywhere…

… we could say after checking Ben Evans's presentation, where he's exploring macro and strategic trends in the tech industry for 2024.

Or we can say that it's still too early and error-prone. Or that we are not sure what we want and what possibilities it provides.

In any case, that is a good analytic, so take your time if you'd like to understand the current state of AI and its near future.

https://www.ben-evans.com/presentations

#ai

CatOps

15 Feb 2024 11:36

You might have heard that the WeaveWorks has gone out of business.

Yet, they did a right thing and open sourced their GitOps Enterprise product, which previously was only accessible to the paid customers.

#cicd #gitops

CatOps

13 Feb 2024 15:52

​​There was an interesting talk at CfgMgmtCamp 2024 about non blocking code reviews.

The good thing is that there’s also an article on this topic from the author

The general idea is that not all the code changes require a code review, especially when there are enough safety nets configured.

As a result, smaller code changes simply sit there and wait for be reviewed, which may take some time, especially in a remote setup.

The solution is to allow such changes as they are and add them to a backlog of pending reviews.

There are more details in the article. Also, here’s a picture from that presentation that kinda captures the spirit of this idea.

#programming #culture #devops

CatOps

07 Feb 2024 12:11

Did you know that Isovalent (a company behind Cilium) has some amazing labs that can teach you about using Cilium, Hubble, and Tetragon.

The labs have multiple tracks, such as: platform, network, security, etc.

These labs also cover topics like the new GatewayAPI. Doing some of these labs tight now at #cfgmgmtcamp24 and love them so far!

#kubernetes #networking #cilium #ebpf

CatOps

05 Feb 2024 12:55

​​A member of our community raises funds for a Mavic 3t for his classmate who serves in the 110 separate mechanized brigade.

You can donate on the Monobank jar

Or on a card directly:

5375 4112 1440 4548

#donations #Ukraine

CatOps

03 Feb 2024 09:36

I’m a CLI guy. For me it was always easier to use good old commands like find and cd to navigate around. At some point I even memorized the flags of tar.

Thus, it is still much easier to me to use plain kubectl with a couple of plugins to navigate the clusters.

However, I know that many folks prefer graphical interfaces or at least some TUI. Also, I remember that a lot of folks were pissed when Lens split into a community and a paid versions.

So, today I want to share a new native Kubernetes desktop client - Seabird.

I haven’t tried it for the reasons I mentioned above, but you may enjoy it.

#kubernetes

CatOps

31 Jan 2024 13:06

How Discord Serves 15-Million Users on One Server.

I think this article was posted before on some of DevOps-related channels.

tl;dr: It's BEAM & Elixir. Also, every time you see an article titled "how a chat app scales to crazy number of users", it's always BEAM.

Yet, in their case, Discord had to make some performance analysis and architectural optimizations with some data offloading. That part is interesting.

P.S. I'd love to learn Elixir someday, but I have zero experience with it or Erlang, or FP in general. So, if you're in the same position, and you're looking for a learning buddy, please, let me know :)

#elixir #architecture

CatOps

29 Jan 2024 12:45

​​For today’s Donations Monday I’d like to share a fundraiser for recon drones by Come Back Alive.

#donations #Ukraine

CatOps

24 Jan 2024 16:44

We don't make posts related to events, unless we are personally invested in those.

Today is one of such examples. I am a part of an organizational committee of the first DOU Technical Conference.

The CfP is still open. However, it's closing tomorrow.

So, don't miss your chance to become a speaker!

The only caveat is that this is an offline event, so you should be able to physically travel to Kyiv on the day of event.

#event

CatOps

22 Jan 2024 11:31

​​My boss from one of my previous jobs used to say: “Stop starting, start finishing”.

TBH, I wish much more companies adopt this motto. So, let’s lead by example!

A joint fundraiser by Come Back Alive and Kyivstar - “We live here” is almost over! Its goal is to raise 175M UAH and right now they are lacking about 4M UAH to close this goal!

So, let’s help them to finish it!

#donations #Ukraine

CatOps

18 Jan 2024 18:08

Google shares their experience from 20 years of SRE.

This article contains overview and lessons learned from some older incidents. It’s nice, but I can’t say that there is any mind blowing information in there. Anyway, it’s nice that it came from Google.

#sre

CatOps

26 Feb 2024 16:08

​​For today’s Donations Monday I would like to remind you about pavlobondarenkopaypal/support-ukrainian-army-with-and-da077ccda388">Pavlo and Naya, who raise money on drones and telecommunication equipment.

P.S. Here’s a photo of a drone you helped to buy last week and it’s already on its way to the place.

CatOps

23 Feb 2024 15:34

Some time ago I had a task to split the helm template output into separate files per object.

So, I found this issue in the Helm’s repository. People were suggesting using AWK for that, but that didn’t work well for me at the time, so I opted out for YQ.

A couple of days ago someone left a comment to that issue that apparently there is a tool called Kubesplit that can do exactly that. So, feel free to use it if you need to achieve something similar to what I did.

#kubernetes

CatOps

20 Feb 2024 10:50

Linux Foundation has some discounts for its courses and certifications till the end of February. Including Kubernetes, ArgoCD, and Istio certifications.

They are still not cheap, but you can save up to 50%, which is nice.

#kubernetes #training #courses

CatOps

18 Feb 2024 19:55

A new issue of the CatOps digest is here! You can find it via:

https://newsletter.catops.dev/p/catops-digest-2024-02-18

#digest #newsletter

CatOps

16 Feb 2024 13:06

Friday is a great day to listen to our latest voice chat!

This time we have discussed what would be the next big thing in operations. And of course, the whole conversation was very AI focused.

You can find the latest episode on:

- YouTube
- Spotify
- Apple Podcasts
- Google Podcasts

#voice_chat #ai

CatOps

14 Feb 2024 14:45

An overview of the progressive delivery principle by Buoyant.

There's also a comparison between Flagger and Argo Rollouts, which are probably the most famous (if not the only) tools designed for progressive delivery.

Now, Buoyant is the company behind Linkerd. So, obviously, their examples are based on Linkerd. However, you don't need any service mesh to adopt this technique. Service mesh will only help you with smarter traffic splitting.

Also, there's an error in this article: it claims that Flagger works with existing deployments, while Argo Rollouts requires its special CRD, which is only partially true. Argo Rollouts can also work with existing Deployment objects. So, in these regards, both tools are the same.

If you prefer a video format, I would also suggest watching a talk about progressive delivery by Carlos Sanchez at Fosdem 2024. There you can see Argo Rollouts in action.

#cicd

CatOps

12 Feb 2024 12:53

​​For today’s Donations Monday I’d like to remind you about Pavlo and Naya, who raise funds for drones and some telecom equipment for our defenders.

Now they have a neat page with all the requisites in one place.

#donations #Ukraine

CatOps

06 Feb 2024 10:51

A delayed CatOps Digest is out!

You can find it here:
https://newsletter.catops.dev/p/catops-digest-2024-02-05

#digest #newsletter

CatOps

04 Feb 2024 10:02

​​Today a new issue of CatOps Newsletter should’ve come out. However, it’s delayed for reasons.

P.S. If you’re attending Fosdem/CfgMgmt Camp, let’s meet for coffee!

CatOps

01 Feb 2024 16:12

And now it's time for a story of how Pinterest scaled to 11 million users with only 6 engineers.

No BEAM this time, though Pinterest isn't a chat app either. They managed to scale that much and stay so lean thanks to the architectural decisions (mostly around keeping it simple).

P.S. Also, Pinterest is probably the best social network out there (if you could call it a social network).

#architecture

CatOps

30 Jan 2024 11:54

There were quite a few nice bundles at HumbleBundle lately!

This one is about cloud as well as ML pipelines design, which should be interesting.

Also, these are O’Reilly books and O’Reilly is famous for their high quality.

#books #bundle

CatOps

25 Jan 2024 13:37

Do you run databases in Kubernetes?

Even if you don't, I bet you may run database migrations there. How do you do that?

This article on "The New Stack" makes a case for GitOps approach to the database migrations in Kubernetes.

*tl;dr*: It's Atlas Operator, there's no alternative.

#kubernetes #databases

CatOps

23 Jan 2024 14:44

For some folks it’s easier to digest information in video format.

I am myself guilty of watching way too many YouTube videos.

Thus, today I want to share with you a bundle of Linux Video Courses from Packt & HumbleBundle.

#books #bundle

CatOps

21 Jan 2024 19:48

A late Sunday digest is out!

https://open.substack.com/pub/catops/p/catops-digest-2024-01-21

#digest #newsletter

CatOps

17 Jan 2024 11:49

We usually don't post event announcements. Yet, there's an exception: our direct involvement in the event :)

On 17th of February I'm speaking at DevOps fwdays'24 conference together with other wonderful people.

You can join the conference either offline in Kyiv or online from anywhere in the World as long as you have the Internet connection.

Also, I have a promo-code CCD4090099 that gives 15% off for any type of tickets!

UPD: if you would like to talk at the conference, you have the last chance to submit your talk via the CfP form!

#event