388
Explore articles, books, news, videos, and insights on software architecture, people management, and leadership. Author: @nelia_loginova
Binary Data Classification
In the previous #aibasics post, I briefly explained the basics of machine learning with Linear Regression. Today let's talk about another type of task - binary data classification. Typical example is determining whether an email is spam or not spam.
Key steps for binary classification:
1. Predict Probability. Take a Logistic Regression model that predict probability (mathematically it returns values between 0 and 1). For example, the probability of an input email being either spam or not spam. If the model predicts 0.72, this means there is a 72% chance the email is spam and 28% chance the email is not spam.
2. Set a Classification Threshold . The classification threshold determines how to assign a binary label (e.g., spam or not spam) based on the predicted probability. For example, the model predicts that a given email has a 75% chance of being spam. Does it mean the email is spam? Actually, no. If the threshold is set at 0.8, then email will be classified as not spam.
3. Evaluate the Model Using a Confusion Matrix. To measure how good our model is, we need to summarize the number of correct and incorrect predictions using confusion matrix:
- True Positive (TP): Correctly predicted positive cases.
- False Negative (FN): Positive cases incorrectly predicted as negative.
- False Positive (FP): Negative cases incorrectly predicted as positive.
- True Negative (TN): Correctly predicted negative cases.
4. Measure Classification Quality. The following metrics are used to define the effectiveness of the result model:
- Accuracy. The proportion of all classifications that were correct, whether positive or negative.
- Recall. The proportion of all actual positives that were classified correctly as positives.
- False Positive Rate. The proportion of all actual negatives that were classified incorrectly as positives.
- Precision. The proportion of all the model's positive classifications that are actually positive
The classification threshold and quality metrics should be adjusted based on the cost of errors for particular domain. If marking important emails as spam is costly, you may increase the threshold to reduce false positives. Conversely, if missing spam emails is more problematic, you may lower the threshold to prioritize catching them.
References:
- Google ML Course: Logistic Regression
- Google ML Course: Classification
- Confusion matrix in machine learning
#aibasics
Today, I’m starting a topic with a picture first.
That's a D. Caruso Mood Map, a tool widely used in emotional intelligence techniques. The tool maps all our emotions on the grid with two scales:
- One scale is for a level of energy (low to high).
- The other scale is for a level of pleasantness (unpleasant to pleasant).
Explanation of how to use that will be shared in the next post 👇
#softskills #leadership
Minimum Viable Architecture
There is no one-size-fits-all architecture for all scales for all project phases. Architecture should evolve with the product and it should be adopted to the requirements at different stages of product lifecycle.
That's the main idea from Randy Shoup talk - Minimum Viable Architecture. He calls this approach “just enough architecture”- the architecture that's good enough for the product to be released at current project stage.
Product Stages and Their Architecture:
📍 Prototyping.
- Goal: proof business concept, test the market and acquire first customers.
- Rapid iterations, a lot of prototyping.
- Technology doesn't matter, use any tools that get results fast.
- No architecture
- Single team
📍 Starting.
- Goal: solve customer needs as cheap as possible, acquire more customers.
- Rapid learning and iterations.
- Use simple, familiar tech stack
- Typically monolith architecture with a single database
- Rely on cloud infrastructure and open-source tools.
- Focus on competency growth, outsource everything else.
- Number of teams grows.
📍 Scaling.
- Goal: stay ahead of rapidly growing business.
- Time to rearchitect: "Getting to rearchitect a system is a sign of success, not failure."
- Build scalable architecture, focus on latency and performance
- Perform migration from monolith to microservices
- Scale team numbers
📍 Optimizing.
- Goal: make a system more sustainable, efficient and effective.
- Focus on small, incremental improvements.
- No major architectural changes.
- Improve operational efficiency.
- Consolidate the teams
I like the idea of matching architecture to business priorities and not overcomplicating the solution on early stages. The talk also shares some tips when rearchitecturing is really needed and how to do it without breaking the existing solution. Some ideas and recommendations about architecture looks too dogmatic for me, but overall the talk is really good ad I recommend to check the full video.
#architecture
Linear Regression
Linear regression is the simplest supervised ML model that finds relationships between features and labels.
Mathematically it looks like:
y'=b+w1*x1 + w2*x2 + ... + wn*xn
y'=b+w1*distance + w2*ride_duration
ML Basic Terms
To be on the same page with AI-experts we need to build a special vocabulary with basic terms and concepts:
✏️ Feature - input parameter for the model. Usually it represents some characteristic of the entity or facts for which the model makes prediction.
✏️ Label - existing answer for input data. Usually used to train supervised models: predicted value can be compared with labels to check the size of discrepancy.
✏️ Loss - the difference between predicted value and label. For different models different functions to calculate loss is used.
✏️ Learning Rate - a floating-point number that tells the optimization algorithm the step size for the iteration while moving toward a minimum of a loss function. If the learning rate is too low, the model can take a long time to converge. If the learning rate is too high, the model may never converge.
#aibasics
I'm introducing a new section on the channel: #aibasics !
Over the past two years, ML is the top trend in the industry with the huge interest not just in tech but across various business domains. ML helps to automate routine tasks and significantly decrease operational costs. And definitely this trend will continue to grow next few years or even more.
As engineers we should at least know the fundamentals of that technology. I mean not just using lots of GenAI tools in daily work but understanding how it works under the hood, its limitations, capabilities and applicability for business and engineering tasks. As of me, I have a significant knowledge gap here, which I plan to close next several months.
I plan to start with the following courses (they are absolutely free):
✏️ Mashing Learning Crash Course from Google that has fresh updates in November
✏️ LLM Course by Cohere
I will use those courses as a base and extend them with additional sources on demand.
So I'm starting my AI learning journey and will share my progress and key takeaways here 💪
Manage Your Energy Level
Recently I wrote about the importance of having high-quality vacations. What I didn’t share is that I went on vacation completely drained, with zero level of internal resources and even a diagnosis from a neurologist 😵💫. It is a tough state to be in, I never want to feel like that again.
So I reflected on how to prevent burning out in the future.
First of all, I understand that's my fault - not heavy work, urgent issues, or company changes. It's primary responsibility of any leader to support their internal resource and energy. That's very important. Leader cannot work without enough energy level, as it's not possible to drive anything or meet business goals in that state.
Next, I started to study different recommendations what to do. The advice is usually very common: exercise, walk, eat well, and have time for hobbies. Unfortunately, I already knew that, but it didn't help me. My issue is that I don't notice the point where I am completely drained and it's too late to go for a walk.
So I need to control internal state somehow. As technical people, we know that to control something we need to measure something. One resource recommends Welltory app that makes personal health analysis based on heart rate variability (Garmin watches have similar features already built-in). Additionally it uses info about sleep, steps, stress level, and more from mostly any smart watch. Looks like a magic, but there is real science under that. This isn’t an ad—just sharing a tool I found useful 🙂.
I've been using the app for about 2 weeks now. The algorithm is still training (about 35% done), but I’m already using its basic features. I periodically make measurements and check overall state: green, orange or red. Based on this, I’ve started taking short recovery breaks at work to avoid hitting zero. Also I control overall health trend to understand if my daily routine needs additional corrections like more exercise, walk, etc.
Burnout is very common problem in our industry that's why I decided to share my experience on what can be helpful to control internal state and support good level of motivation and energy. Of course, 2 weeks are not enough to say the approach works. Put likes if the topic is interesting and I'll share my results in 1-2 months.
Stay healthy and take care of yourself.
#softskills #productivity
Uber’s Gen AI On-Call Copilot
GenAI continues its march in routine automation. This time Uber shared their experience with Genie - on-call support automation for internal teams.
The issue is very common for large companies with many teams: there is some channels (for Uber, it's slack with ~45 000 questions per month) where teams can put questions and request help with the service or technology. Of course, there are a lot of docs and relevant articles, but they are fragmented and spread across internal resources. It's really hard for users to find answers on their own. As a result, the number of repetitive questions grows, load and demand on support engineers increase.
Key elements of implemented solution:
✏️ RAG (Retrieval-Augmented Generation) Approach to work with LLM
✏️ Data Pipeline: Information from wikis, internal Stack Overflow, and engineering docs is scraped daily, transformed into vectors, and stored in an in-house vector database with the source links. Data pipeline is implemented on Apache Spark.
✏️ Knowledge Service: When a user posts a question in Slack, Genie’s backend converts it into a vector and fetches the most relevant chunks from the vector database.
✏️ User Feedback: Users can rank answers as Resolved, Helpful, Not Helpful, or Relevant, these ratings are used to analyze answer quality.
✏️ Source Quality Improvements: There is a separate evaluation process to improve source data quality. The LLM performs docs analysis and returns an evaluation score, explanations of the score and actionable suggestions to improve. All these information is collected to an evaluation report for further analysis and fixes.
Since Genie’s launch in September 2023, Uber reports it has answered 70,000 questions with 48.9% helpfulness rate, saving 13 000 engineering hours😲. It's impressive! I definitely want to have something similar at my work. Just a small hurdle left—get the budget and resources for implementation. No big deal, right? 😉
#engineering #usecase #ai
Take a Vacation
Last week I was on vacation, so there was a little break in the publications😌. Therefore I would like to talk a little about the vacation and how important it is. High quality vacation is not just opportunity for relax but it is also a prevention mechanism for many serious diseases.
But it’s not enough just to take vacations regularly; the way you spend them determines if you re-charge your internal battery or not.
My tips for a good vacation:
✏️ Take Enough Time: Ideally, a vacation length should be at least 14 days (as a single period). If you feel heavily exhausted, then better to take 21 days. That time is usually enough to recharge.
✏️ Change the Scenery: Travelling to a new place (even a short trip) gives you new impressions, experience, fill you with new ideas, inspiration and energy. Spending time outside standard surroundings significantly decreases an overall strain level. The fact is also proved by German researchers.
✏️ Digital Detox: Don't touch your laptop, don't open working chats, don't read the news, minimize social networks usage. Give the rest to your brain from constant information noise.
✏️ Be Spontaneous: Don't try to plan everything: constant following the schedule makes vacation feel more work-like and doesn't allow to enjoy the moment. Spontaneous activities can provide more fun and satisfaction.
✏️ Do Nothing: Allow yourself to take time for idleness. That's really difficult as you feel just wasting time that can be spend more effectively😀. But that's the trick: state of nothingness rewires the brain, improve creativity and problem solving capabilities.
So take care of yourself and plan a proper rest during the year.
#softskills #productivity
SMURF Testing
Google introduced new mnemonic for test quality attributes - SMURF:
📌 Speed: Unit tests are faster than other test types so they can be run more often.
📌 Maintainability: Cost of debugging and maintaining tests.
📌 Utilization: A good test suite optimizes resource utilization, fewer resources cost less to run.
📌 Reliability: Sorting out flaky tests wastes developer time and costs resources in rerunning the tests.
📌 Fidelity: High-fidelity tests come closer to approximating real operating conditions. So integration tests have more fidelity than unit tests.
In many cases improving one quality attribute can affect the others, so be careful and measure your costs and trade-offs.
#engineering #testing
API Governance
In modern distributed systems, where individual teams manage different services, it's pretty common for each team to create their own APIs in different ways. Each team tries to make their APIs unique and impressive. As a result, a company may have a lot of APIs that follow different rules and principles and reflect organizational structure instead of business domains (you remember Conway's Law, right?). This can be a full mess.
To avoid this, APIs must be properly managed to stay consistent.
API Governance is the set of practices, tools and policies to enforce API quality, security, consistency and compliance. It involves creating standards and processes for every stage in the API lifecycle, from design, development, and testing to deployment, management, and retirement.
API Governance consists of the following elements:
📍 Centralization. A single point where policies are created and enforced.
📍 API Contract. Standard specifications to define APIs like OpenAPI, gRPC, GraphQL, AsyncAPI and others.
📍Implementation Guidelines. Establish and enforce style guidelines for all APIs. Good examples are Google Cloud API Guidelines , Azure API Guidelines.
📍 Security Policies. Defining API security standards and policies that protect sensitive data from cyber threats and ensuring API compliance with regulations.
📍 Automation. Developers and other roles need to quickly make sure that APIs are compliant with the enterprise standards at various stages of the lifecycle.
📍Versioning.
📍Deprecation Policy.
📍API Discovery. Provide a way to easily search for and discover existing APIs.
API Governance provides the guardrails to develop high-quality consistent APIs within the company. But to make it work a good level of automation is required.
#engineering #api
Shipping Threads in 5 Months
As developers, we often prefer writing new code rather then reusing the older one. Of course, new code will be better, faster, more readable, maintainable, it will use newer tools and frameworks and definitely will be great. Spoiler: No 😉
Old battle-tested code can be significantly better because it's already tested, covered by automations, it has common features in place, little learning encoded, it doesn't have stupid mistakes already, it's mature.
That's why I think there is really interesting experience that shared by Meta team how they reuse Instagram code to build Threads app from scratch.
Initially, the team had an ambitious goal to build a service to compete with X (Twitter) in a couple of months. To achieve that, they decided to use existing Instagram code with core features like profiles, posts, recommendations, followers as the base for the new service. Additionally, Threads was built on existing Instagram infra with support of existing product teams. This approach allows to deliver new fully-featured service in 5 months.
Key findings during the process:
✏️ You need the deep knowledge of legacy code to successfully reuse it
✏️ Code readability really matters
✏️ Repurposing and customization existing code for the new requirements brough additional technical debt that should be paid in the future
✏️ Shared infrastructure and product teams can significantly reduce development costs
✏️ Old code is already tested in real conditions and contains less issues then the new code
So don't rush to rewrite the existing code. Thoughtful evolutionary approach can bring more business benefits, reducing time to market and overall development costs.
#engineering #usecase
Cannot stop writing about Pixar. Accidentally open non-existing-page. That's the best 404 page I've seen 😍 .
Читать полностью…
True Inspiration from Pixar. Part 2: Protect New Ideas
That's the second part of Creativity, Inc.: Overcoming the Unseen Forces That Stand in the Way of True Inspiration book overview (First part is there). It is about practices that can help to protect new ideas from bureaucracy, fear of feedback, and thinking "experienced people know best."
To support creativity and innovation, Pixar's leaders built a culture based on the following principles:
✏️ Candor. Ask people about candor not honesty, build processes that demonstrate value of the candor on all levels. Pixar uses Braintrust practice to train that:
- Regular offline meetings every 2-3 months.
- Teams present film fragments to identify issues.
- There is no gradation by official titles, everyone's opinion is important, there is no stupid or destructive feedback.
- Feedback should be focused on the problem not the person.
- Criticism is part of improving the work, not as competition.
- Create an atmosphere of trust where all member are interested in great results.
✏️ Value of Failures. Don’t fear the failure. Failure is a chance to learn. Start working, get feedback, learn, and try again. The author says that Pixar culture is unique as it doesn't just allow the people to make mistakes, but it expects them to make those mistakes.
✏️ Protect Ugly Babies. New ideas are usually not beautiful, so author called them the ugly babies. It takes time and patience for them to grow and shine. All Pixar films started as simple, sometimes awkward ideas, going through many iterations. It's important to protect new ideas from conservatism, habits to do something by only known ways, because it leads to predictable but mediocre results, killing true inspiration.
✏️ Change and Randomness. People don't like changes because it feels unsafe or overwhelming. Asking "What if?" question helps teams imagine possibilities and break through the fear barriers. One more important note there is about rules. Rules appear by some reason, but reasons are changed over time and can be not actual anymore. Outdated rules improves bureaucracy and kill creative atmosphere.
✏️ The Hidden. True leaders accept that employees often have a deeper understanding of problems. Managers don’t need to know everything, but they must encourage open communication to get a bigger picture of what’s happening in teams. Healthy cultures encourage employees share theirs opinions, report problems, make suggestions. Otherwise leaders can be in dangerous information isolation.
To be continued....
#booknook #softskills #leadership
Book covers for Creativity, Inc.: Overcoming the Unseen Forces That Stand in the Way of True Inspiration
#booknook
The Mood Map
Let's check what is interesting in that tool and how it can be used in our daily life and work.
✏️ Emotions Mapping. That's ability to recognize emotions in yourself, your colleagues, and your partners. One of the helpful techniques is mirroring — matching the other person’s speech rate and tone of voice. On the neurophysiology level it means “This person is like me!”, that makes communication more pleasant, gives the sense of safety and increases the chances to reach agreements.
✏️ Task Selection. The Mood Map helps to choose tasks for yourself or your team based on current emotional states. For example, anxiety can sharpen focus, happiness and joy are good for creativity, contentment improves chances to come to a consensus. The key idea is to either pick tasks that match your current state or shift your state to suit the task. This applies to the teams too: "If you have a brainstorming session and the team seems anxious, that’s not a good match. As a leader, you either have to change the tone of the room or change the agenda to match the tone".
✏️ Understanding. What makes you happy might not make someone else happy That's important to understand and learn what motivates and inspires your team members. At the same time, emotions have universal reasons. If you understand the root of someone’s behavior, you can address it effectively.
✏️ Changing Emotions. Agreements are hard to reach if you and the other person are in different quadrants of the Mood Map. Ideally, everyone needs to move to the `green` quadrant to reach a consensus. However, jumping directly from `red` to green is almost impossible. Instead, you can guide someone through smaller transitions, like red -> blue -> green. For example, if someone is in the red quadrant, speaking slowly and calmly can help reduce emotional intensity and shift them toward blue.
I used to be skeptical about emotional intelligence techniques, but that tool looks to be helpful and practical.
Additional trick there: during complex conversations, if emotions are escalating, pause and ask yourself, “What am I feeling right now? Why?”. Reflection helps to shift your brain from the emotional side to the logical side. Once you’re back in a logical state, you can better manage the situation and improve your chances of success.
References:
- davidcaruso4617/videos">David Caruso Youtube Channel
- Can emotional intelligence be learned?
- Emotional Intelligence in a Changing World
#softskills #leadership #communications #productivity
Visualized summary for main stages from the talk Minimum Viable Architecture.
#architecture
Visualization of how different loss functions can change model training results. As mentioned above MSE moves the model more toward the outliers, while MAE doesn't.
#aibasics
Examples of a model that converges vs one that doesn't.
#aibasics
ML Introduction
Let's start AI basics with the ML definition and their types.
Definition from Google ML Introduction course:
ML is the process of training a piece of software, called a model, to make useful predictions or generate content from data.
Cloud Ecosystem Trends
This week CNCF published Emerging trends in the cloud native ecosystem with a list of trends that will continue to grow in 2025.
Top trends:
🚀 Cloud Cost Optimizations. With growing cloud adoption, businesses focus on controlling cloud costs using tools like Karpenter and OpenCost. The same trend was also highlighted by FinOps Foundation earlier this year.
🚀 Platform Engineering (I did overview there). Extend developer experience with platforms for observability, policies as a code, internal developer portals, security, CI/CD, and storage to speed up business development.
🚀 AI Synergy. The trend is to support AI training and operations in the cloud. New actively developed projects there:
- OPEA: a collection of cloud-native patterns for GenAI workloads
- Milvus: a high-performance vector database
- Kubeflow: a project to deploy machine-learning workflows on Kubernetes
- KServe: a toolset for serving predictive and generative machine-learning models
🚀 Observability Standards Unification. Projects like OpenTelemetry and the Observability TAG unify standards, minimize vendor locks, and reduce costs.
🚀 Security. Security is a top priority topic in CNCF. There are some newly graduated projects in that area (like Falco) and separate TAG-Security group that publishes white papers that offer directions to the industry on the topic of security.
🚀 Sustainability (more about GreenOps there). Sustainability tools (like Kepler, OpenCost) measure carbon footprints of Kubernetes applications. The area is under active development now, but it already has promising open-source projects and standards.
It's interesting that overprovisioning and high resource waste is still the main problem in modern clouds. According to the Kubernetes Cost Benchmark Report clusters with 50 or more CPUs used only 13% of their provisioned capacity, memory utilization was at the level of 20%. This shows a huge opportunity for future optimizations.
#news
Columnar Databases
Traditional databases store data in a row-oriented approach that is optimized for transactional, single-entity data lookup. But if you need to aggregate data by a specific column, the system has to read all columns from disk, which slows down query performance and increase resource usage.
To solve the issue, columnar databases was introduced.
Columnar database is a type of a database that stores data in columns together on the disk.
Imagine the following sample:
|Account|LastName|FirstName|Purchase,$|
| 0122 | Jones | Jason | 325.5 |
| 0123 | Diamond| Richard | 500 |
| 0124 | Tailor | Alice | 125 |
0122, Jones, Jason, 325.5;
0123, Diamond, Richard, 500;
0124, Tailor, Alice, 125;
0122, 0123, 0124;
Jones, Diamond, Tailor;
Jason, Richard, Alice;
325.5, 500, 125;
Google ARM Processor
Last week, Google announced their own custom ARM-based processor for general-purpose workloads. They promised up to 65% better price-performance and up to 60% better energy-efficiency.
Why is it interesting? Until now, only AWS offered a custom cost-optimized ARM processor - AWS Graviton. And now Google joined the competition. This shows that interest in ARM processors still grows and continue to grow in the future.
From engineering perspective, it's not possible just to switch workload from one architecture to another as images need to be pre-built for a specific architecture. One of the ways to test ARM nodes and migrate smoothly on the new architecture is by using multi-architecture images (I wrote about that here)
#engineering #news
API Governance at Scale
The most difficult part of API Governance is to ensure that developers follow provided guidelines and policies. Without proper controls, the real code will eventually drift from the guidelines—it’s only a matter of time. This doesn’t happen because developers are bad or unwilling to follow the rules, but because we’re human, and humans make mistakes. Mistakes accumulate and grow over time and as a result you can get APIs that are too far from initial recommendations.
In small teams with a small codebase, developers education can work, and trained reviewers can ensure the code follows the rules. But as your team or organization grows, this approach isn't enough. I strongly believe that only automation can maintain policy compliance over a large codebase or multiple teams.
Google recently published API Governance at Scale, sharing their experience and tools to control API guidelines execution.
They introduced 3 key components:
✏️ API Improvement Proposals (AIPs). This is a design document providing high-level documentation for API development. Each rule is introduced as a separate AIP that consists of a problem description and a guideline to follow (Example, AIP-126).
✏️ API Linter . This tool provides real-time checks for compliance with existing AIPs.
✏️ API Readability Program. This is an educational program to prepare and certify API design experts, who then perform a code review for API changes.
While Google developed the AIPs concept, they encourage other companies to adopt the approach. Many of the rules are generic and easily reusable. They even provide a special guide on how to adopt AIPs. Adoption strategy is not finished now, but preparation status can be tracked via appropriate Github issue.
#engineering #api
API Governance: Versioning
Let's continue today with API management and talk about versioning. To define you versioning policy you need to answer the following questions:
😲 What versioning method will be used?
🤔 When do you need to release a new version?
😬 What naming convention to use?
😵💫 How to keep compatibility with the clients?
The most popular versioning strategies:
✏️ No Versioning. Yes, that's also a choice 😀
✏️ Semantic Versioning. It's well-know strategy to version anything in software development world.
✏️ Stability Levels: alpha, beta, stable. Major version is changed on breaking changes. Examples: v1alpha, v2beta, v1aplha3, v2. More details in Google API Design Guide.
✏️ Release Numbers: Simple sequential versions like v1, v2, v3, updated mainly for breaking changes.
✏️ Product Release Version: Use your product’s version for APIs. Example: product version 2024.3 then API version 2024.3. In that case version is changed even if there are no major changes, but it really simplifies tracking compatibility between releases and APIs.
To reduce the impact of API changes the following compatibility strategies can be used:
✏️ Synchronized Updates: Both API and clients are updated and delivered together. Simple, fragile. It can be useful if you control and manage all API consumers.
✏️ Client Supports Multiple Versions: One client can work with multiple API versions, but outdated clients may stop working and require updates to match newer APIs.
✏️ API Serves Multiple Versions: New API version is added in parallel to the existing one on the same server. In that case you may serve as many versions as you need to support all your clients. To reduce API management overhead Hub-Spoke pattern can be used: the hub represents the main version, while spoke versions are converted from the hub. This approach is actively used in Kubernetes, so you can read more details in Kubebuilder Conversion Concepts.
Analyze your requirements and architecture, set clear rules, define the versioning and compatibility approach. It's really important to document those decisions and socialize them to your clients.
#engineering #api
Put Your Own Mask On First
"Put your own mask on first before assisting your child 😷"—they always say it on the plane before the flight. It sounds clear and familiar for us. But this same rule can apply to other parts of our lives. Metaphorically of course.
What do we usually do to meet deadlines, deal with a pile of issues at work, or handle business pressures? The most common scenario is to work more and more, eventually leading to burnout. But a burned-out leader can't solve problems effectively or help their team to survive in the storm of difficulties or achieve business goals.
It may sound counterintuitive but more pressure and problems you face at work, the more time and care you need to give yourself: proper nutrition, walking, regular physical activity, full sleep and less overwork.
When you take care of yourself, you can better take care of your team. So put your own mask on first before assisting to others.
#softskills #leadership
Observability 2.0
There is very charismatic talk from Charity Majors, Co-Author of "Observability Engineering", called Is it time for Observability 2.0? Sounds intriguing, so let's check what's inside.
Key ideas:
📍 Observability 1.0. It's traditional observability techniques based on 3 pillars: logs, metrics and traces. It's complex, expensive and requires skilled engineer to analyze correlations between different sources of data.
📍 Observability 1.0 has significant limitations:
- static dashboards
- lack of important details
- designed to answer to pre-defined questions
- multiple source of truth
- multiple systems to support
📍Observability 2.0 paradigm is based on the idea to use wide structured logs that contain all necessary information. It makes easy to aggregate data, zoom in and out for details when needed.
📍Observability 2.0 based on single source of truth - logs, all others is just visualization, aggregation and making dynamic queries. There is no data duplication. There is no need to install and maintain a set of tools for the telemetry. In that terms it's cheaper.
Implementation tips:
📍 Instrument the code using principles of Canonical Logs (we already checked the concept there)
📍Add Trace IDs and Span IDs to trace requests chain execution
📍 Feed the data into a columnar store, to move away from predefined schemas or indexes
📍 Use a storage engine that supports high cardinality
📍Adopt tools with explorable interfaces or dynamic dashboards
Our systems have become too hard and complicated. So it's critical to have effective observability tools and practices. The approach from the talk looks promising especially as it doesn't require any new tools to be developed. Let's see if it become a new trend in observability implementation.
#engineering #observability
True Inspiration from Pixar. Part 3: Broadening the View
That's the third part of Creativity, Inc.: Overcoming the Unseen Forces That Stand in the Way of True Inspiration book overview (Other parts: 1, 2).
Throughout the book the author shares his management ideas and principles. There is one though that I find very interesting from practical point if view. He says that people who live or work together tend to become closer and share similar mental models and behavior patterns (event if those models are wrong). That fact can be used to build right team culture and create atmosphere of creativity and innovation.
Practical tools to improve collaboration within the team or company:
✏️ Daily Meetings. Daily information exchange and team work analysis improves overall team productivity.
✏️ Research Trips. All our mental models are wrong, we need regularly clean our believes and get rid of cliches. Pixar sends their employees to the locations that are relevant to the topics of the movies they are working on (e.g. real student campus for Monster University movie to get better understanding of students life and environment).
✏️ Power of Restrictions. Limited resources help to focus on important things, improve decision making and optimize internal processes.
✏️ Technology and Art Integration. New tech tools should be actively used to automate the routine, freeing up time for more important and creative tasks.
✏️ Short Experiments. Perform short experiments to prove new ideas.
✏️ Train the Vision. Pixar offers drawing courses to all employees, because drawing improves observation skills, stimulates the right side of the brain activity that is responsible for human creativity.
✏️ Dissection. It's very similar to Agile retrospective. After a film release, teams collect their lessons learnt, aggregate good experience and discuss the mistakes. The author suggest the following practice: ask everyone to list 5 things they'd do again and 5 they wouldn’t.
✏️ Continuous Education. Pixar encourages constant learning across fields. Pixar University has classes for drawing, dancing, acting skills and others. Employees from different departments attend classes together, establishing connections outside of work roles. One more important idea behind the practice: if you do something that you don't usually do at work you keep your brain healthy.
The main message from the author to the leaders: people are the most valuable part of any company🫶. Invest time and effort in your teams, help them reveal their potential, trust your colleagues, and avoid trying to control everything, delegate. Encourage openness and trust, create a safe environment within your organization. It’s hard, ongoing work, but it's the only way to build and maintain a creative culture.
I grew up on Pixar movies, and I still go to the cinema for every new release. It was really interesting to read how it's organized inside. Surprisingly, film making is very similar to software development: like any project it has budget and resource restrictions, it starts from MVP, then there is set of iterations and experiments to build the product, teams have daily syncups, some standard project processes, multiple "release candidate", deadlines and, of course, restrospectives. So suggested practices and ideas are also applicable for our daily routine to improve creativity and collaboration.
#booknook #softskills #leadership
GenAI for Legacy Systems Modernization
While most people actively write about using GenAI tools to generate new code, there is a new Thoughtworks publication that focuses on the opposite — using AI to understand and refactor legacy systems.
What makes legacy systems modernization expensive?
- Lack of design and implementation details knowledge
- Lack of actual documentation
- Lack of automated tests
- Absence of human experts
- Difficulty to measure the impact of the change
To address these challenges Thoughtworks team developed a tool called CodeConcise. But the authors highlighted that you don't need exactly this tool, the approach and ideas can be used as a reference to implement your own solution.
Key concepts:
✏️ Treat code as data
✏️ Build Abstract Syntax Trees (ASTs) to identify entities and relationships in the code
✏️ Store these ASTs in graph database (neo4j)
✏️ Use a comprehension pipeline that traverses the graph using multiple algorithms, such as Depth-first Search with backtracking in post-order traversal, to enrich the graph with LLM-generated explanations at various depths (e.g. methods, classes, packages)
✏️ Integrate the enriched graph with a frontend application that implements Retrieval-Augmented Generation (RAG) approach
✏️ The RAG retrieval component pulls nodes relevant to the user’s prompt, while the LLM further traverses the graph to gather more information from their neighboring nodes to provide the LLM-generated explanations at various levels of abstraction
✏️ The same enrichment pipeline can be used to generate documentation for the existing system
For now the tool was tested with several clients to generate explanations for low-level legacy code. The next goal is to improve the model to provide answers at the higher level of abstraction, keeping in mind that it might not be directly possible by examining the code alone.
The work looks promising and could significantly reduce the time and cost of modernizing old systems (especially written on exotic languages like COBOL). It simplifies reverse-engineering and helps generate knowledge about the current system. The authors also promised to share results on improving the current model and provide more real life examples for the tool usage.
#news #engineering #ai
True Inspiration from Pixar. Part 1: Core Values
One of the recent books that really inspired me is Creativity, Inc.: Overcoming the Unseen Forces That Stand in the Way of True Inspiration by Ed Catmull, co-founder of Pixar Animation. It’s presented as a book about creative leadership and management principles that helped Pixar to build unique culture and become the best animation studio in the world. But it's much more than that.
What makes this book special for me? It's not just practical advice, but the story of a long way for a dream came true.
Book starts with Pixar’s history. Did you know that Pixar originally began with creating graphic design programs and selling special computers for them? From a young age, Ed Catmull dreamed about making computer-animated feature films (Disney films were all hand-drawn). But no one believed in his idea, so Pixar was focused on improving how computers processed and displayed graphic data.
Things changed in 1977 with the release of Star Wars by George Lucas. The era of computer effects started. For a few years, Pixar was a part of Lucasfilm, but the dream of making a fully animated movie wasn’t achieved.
In 1986, Steve Jobs bought Pixar, and the team finally had the chance to work on their first film. Toy Story in 1995 became a Pixar triumph. It was first fully computer-animated film in the world, it was extremely successful and it really changed the animation industry. But it was only the beginning. It took a lot of efforts to create next movies with the same quality, scale company culture, support creativity, keep company core values with its growth. Today, Pixar is part of Disney but operates as a separate division with its own vision, ideas, and projects.
Lessons learnt from the growth:
✏️ Building the right team is the foundation for success.
✏️ Focus on teamwork, not on individual talents.
✏️ Focus on the people, their habits, values, help them to reveal their talents.
✏️ People are more important than ideas because people create ideas.
✏️ Quality must be a set as a main condition before project start.
To be continued....
#booknook #softskills #leadership