14849
Top stories from https://news.ycombinator.com (with 100+ score) Contribute to the development here: https://github.com/phil-r/hackernewsbot Also check https://t.me/designer_news Contacts: @philr
The XOR Texture (2004) (Score: 151+ in 12 hours)
Link: https://readhacker.news/s/6jHr7
Comments: https://readhacker.news/c/6jHr7
Ergo Chat – A modern IRC server written in Go (Score: 150+ in 9 hours)
Link: https://readhacker.news/s/6jHrr
Comments: https://readhacker.news/c/6jHrr
Show HN: I built an open-source data pipeline tool in Go (Score: 150+ in 13 hours)
Link: https://readhacker.news/s/6jG5n
Comments: https://readhacker.news/c/6jG5n
Every data pipeline job I had to tackle required quite a few components to set up:
- One tool to ingest data
- Another one to transform it
- If you wanted to run Python, set up an orchestrator
- If you need to check the data, a data quality tool
Let alone this being hard to set up and taking time, it is also pretty high-maintenance. I had to do a lot of infra work, and while this being billable hours for me I didn’t enjoy the work at all. For some parts of it, there were nice solutions like dbt, but in the end for an end-to-end workflow, it didn’t work. That’s why I decided to build an end-to-end solution that could take care of data ingestion, transformation, and Python stuff. Initially, it was just for our own usage, but in the end, we thought this could be a useful tool for everyone.
In its core, Bruin is a data framework that consists of a CLI application written in Golang, and a VS Code extension that supports it with a local UI.
Bruin supports quite a few stuff:
- Data ingestion using ingestr (https://github.com/bruin-data/ingestr)
- Data transformation in SQL & Python, similar to dbt
- Python env management using uv
- Built-in data quality checks
- Secrets management
- Query validation & SQL parsing
- Built-in templates for common scenarios, e.g. Shopify, Notion, Gorgias, BigQuery, etc
This means that you can write end-to-end pipelines within the same framework and get it running with a single command. You can run it on your own computer, on GitHub Actions, or in an EC2 instance somewhere. Using the templates, you can also have ready-to-go pipelines with modeled data for your data warehouse in seconds.
It includes an open-source VS Code extension as well, which allows working with the data pipelines locally, in a more visual way. The resulting changes are all in code, which means everything is version-controlled regardless, it just adds a nice layer.
Bruin can run SQL, Python, and data ingestion workflows, as well as quality checks. For Python stuff, we use the awesome (and it really is awesome!) uv under the hood, install dependencies in an isolated environment, and install and manage the Python versions locally, all in a cross-platform way. Then in order to manage data uploads to the data warehouse, it uses dlt under the hood to upload the data to the destination. It also uses Arrow’s memory-mapped files to easily access the data between the processes before uploading them to the destination.
We went with Golang because of its speed and strong concurrency primitives, but more importantly, I knew Go better than the other languages available to me and I enjoy writing Go, so there’s also that.
We had a small pool of beta testers for quite some time and I am really excited to launch Bruin CLI to the rest of the world and get feedback from you all. I know it is not often to build data tooling in Go but I believe we found ourselves in a nice spot in terms of features, speed, and stability.
https://github.com/bruin-data/bruin
I’d love to hear your feedback and learn more about how we can make data pipelines easier and better to work with, looking forward to your thoughts!
Best, Burak
Droste’s Lair (Score: 150+ in 1 day)
Link: https://readhacker.news/s/6jDBD
Comments: https://readhacker.news/c/6jDBD
FTC bans hidden junk fees in hotel, event ticket prices (🔥 Score: 165+ in 1 hour)
Link: https://readhacker.news/s/6jGM7
Comments: https://readhacker.news/c/6jGM7
Crunch – a Scheme compiler with a minimal runtime (Score: 150+ in 9 hours)
Link: https://readhacker.news/s/6jFqR
Comments: https://readhacker.news/c/6jFqR
How I used linear algebra to build an interactive diagramming editor (Score: 150+ in 12 hours)
Link: https://readhacker.news/s/6jEM9
Comments: https://readhacker.news/c/6jEM9
Advent of Code on the Nintendo DS (Score: 150+ in 18 hours)
Link: https://readhacker.news/s/6jE3A
Comments: https://readhacker.news/c/6jE3A
Getting to Two Million Users as a One Woman Dev Team (🔥 Score: 162+ in 1 hour)
Link: https://readhacker.news/s/6jFAX
Comments: https://readhacker.news/c/6jFAX
Running NetBSD on IBM ThinkPad 380Z (Score: 150+ in 6 hours)
Link: https://readhacker.news/s/6jEF9
Comments: https://readhacker.news/c/6jEF9
Always go to the funeral (2005) (Score: 150+ in 10 hours)
Link: https://readhacker.news/s/6jDTe
Comments: https://readhacker.news/c/6jDTe
Mirror bacteria research poses significant risks, scientists warn (❄️ Score: 151+ in 3 days)
Link: https://readhacker.news/s/6jv6b
Comments: https://readhacker.news/c/6jv6b
New LLM optimization technique slashes memory costs up to 75% (❄️ Score: 150+ in 3 days)
Link: https://readhacker.news/s/6jw4B
Comments: https://readhacker.news/c/6jw4B
Making a watch from scratch (❄️ Score: 154+ in 2 days)
Link: https://readhacker.news/s/6jxWP
Comments: https://readhacker.news/c/6jxWP
Ruby Video – On a mission to index all Ruby conferences (Score: 150+ in 1 day)
Link: https://readhacker.news/s/6jzQ7
Comments: https://readhacker.news/c/6jzQ7
Voxon: Real time interactive volumetric holograms (❄️ Score: 150+ in 4 days)
Link: https://readhacker.news/s/6jwKg
Comments: https://readhacker.news/c/6jwKg
The Headlight Brightness Wars (Score: 150+ in 12 hours)
Link: https://readhacker.news/s/6jGfY
Comments: https://readhacker.news/c/6jGfY
Natural Number Game: build the basic theory of the natural numbers from scratch (Score: 150+ in 19 hours)
Link: https://readhacker.news/s/6jFbs
Comments: https://readhacker.news/c/6jFbs
After 3 Years, I Failed. Here's All My Startup's Code (Score: 150+ in 13 hours)
Link: https://readhacker.news/s/6jFsY
Comments: https://readhacker.news/c/6jFsY
3D-Printed Dune Chess Set (❄️ Score: 150+ in 2 days)
Link: https://readhacker.news/s/6jAmA
Comments: https://readhacker.news/c/6jAmA
Microsoft Confirms Password Deletion for 1B Users (❄️ Score: 150+ in 3 days)
Link: https://readhacker.news/s/6jwJD
Comments: https://readhacker.news/c/6jwJD
Moon (🔥 Score: 197+ in 43 minutes)
Link: https://readhacker.news/s/6jGcP
Comments: https://readhacker.news/c/6jGcP
MIT study explains why laws are written in an incomprehensible style (Score: 152+ in 12 hours)
Link: https://readhacker.news/s/6jEAz
Comments: https://readhacker.news/c/6jEAz
Our muscles will atrophy as we climb the Kardashev Scale (Score: 150+ in 20 hours)
Link: https://readhacker.news/s/6jCZP
Comments: https://readhacker.news/c/6jCZP
Using Guile for Emacs (Score: 150+ in 19 hours)
Link: https://readhacker.news/s/6jCCn
Comments: https://readhacker.news/c/6jCCn
Waymo will bring autonomous vehicles to Tokyo (🔥 Score: 150+ in 3 hours)
Link: https://readhacker.news/s/6jExB
Comments: https://readhacker.news/c/6jExB
Load is not what you should balance: Introducing Prequal (Score: 150+ in 22 hours)
Link: https://readhacker.news/s/6jBq5
Comments: https://readhacker.news/c/6jBq5
UK's Online Safety Act comes into force (Score: 150+ in 18 hours)
Link: https://readhacker.news/s/6jBFx
Comments: https://readhacker.news/c/6jBFx
The Antikythera mechanism – 254:19 ratio (Score: 150+ in 1 day)
Link: https://readhacker.news/s/6jArM
Comments: https://readhacker.news/c/6jArM
Go Protobuf: The New Opaque API (Score: 151+ in 4 hours)
Link: https://readhacker.news/s/6jDyV
Comments: https://readhacker.news/c/6jDyV