Big Data Science

09 Oct 2024 17:59

⚡️HTTP SQLite StarbaseDB

StarbaseDB is a powerful and scalable open source database that is based on SQLite and runs over the HTTP protocol. This database is built to run in a cloud environment (e.g. on Cloudflare), allowing it to scale efficiently down to zero based on load. Key benefits of StarbaseDB include:

✅Ease of use: Provides the ability to work through HTTP requests, making it easy to integrate with various systems and services.
✅Scalability: Automatically adjusts to load volume with the ability to scale both ways.
✅Support for SQLite: Utilize the time-tested and lightweight SQLite database for data storage.
✅Open Source: Open source, allowing developers to customize and improve the system to suit their needs.

It is suitable for developers who are looking for a simple and reliable way to organize databases with minimal customization and high availability in cloud platforms such as Cloudflare.

Big Data Science

02 Oct 2024 17:59

💡😎3 unpopular but very necessary visualization libraries

Supertree is a Python library designed for interactive and convenient visualization of decision trees in Jupyter Notebooks, Jupyter Lab, Google Colab and other notebooks that support HTML rendering. With this tool, you can not only visualize decision trees, but also interact with them directly in the notebook.

Mycelium is a library for creating graphical visualizations of machine learning models or any other directed acyclic graphs. It also provides the ability to use the Talaria graph viewer to visualize and optimize models

TensorHue is a Python library designed to visualize tensors directly in the console, making it easier to analyze and debug them, making the process of working with tensors more visual and understandable.

Big Data Science

27 Sep 2024 17:59

😎💡🔥A selection of unpopular but very useful Python libraries for working with data

Bottleneck is a library that speeds up NumPy methods up to 25 times, especially when processing arrays containing NaN values. It optimizes calculations such as finding minima, maxima, medians, and other aggregate functions. By using specialized algorithms and handling missing data, Bottleneck significantly speeds up work with large data sets, making it more efficient than standard NumPy methods.

Nbcommands is a tool that simplifies code search in Jupyter notebooks, eliminating the need for users to search manually. It allows you to find and manage code by keywords, functions, or other elements, which significantly speeds up working with large projects in Jupyter and helps users navigate their notes and code blocks more efficiently.

SciencePlots is a style library for matplotlib that allows you to create professional graphs for presentations, research papers, and other scientific publications. It offers a set of predefined styles that meet the requirements for data visualization in scientific papers, making graphs more readable and aesthetically pleasing. SciencePlots makes it easy to create high-quality graphs that meet the standards of academic publications and presentations.

Aquarel is a library that adds additional styles to visualizations in matplotlib. It allows you to improve the appearance of graphs, making them more attractive and professional. Aquarel simplifies the creation of custom styles, helping users create graphs with more interesting designs without having to manually configure all the visualization parameters.

Modelstore is a library for managing and tracking machine learning models. It helps organize, save, and version models, as well as track their lifecycle. With Modelstore, users can easily save models to various storages (S3, GCP, Azure, and others), manage their updates and restore. This makes it easier to deploy and monitor models in production environments, making working with models more convenient and controllable.

CleverCSV is a library that improves the process of parsing CSV files and helps avoid errors when reading them with Pandas. It automatically detects the correct delimiters and format of CSV files, which is especially useful when working with files that have non-standard or heterogeneous structures. CleverCSV simplifies working with data by eliminating errors associated with incorrect recognition of delimiters and other file format parameters.

Big Data Science

23 Sep 2024 17:59

📊Quick Tips for Handling Large Datasets in Google's Pandas

Pandas is a great tool for working with small datasets, typically between two and three gigabytes in size.

For datasets larger than this threshold, using Pandas is not recommended. This is because if the dataset size exceeds the available RAM, Pandas loads the entire dataset into memory before processing. Memory issues can arise even with smaller datasets, as preprocessing and rewriting create duplicate DataFrames.

⚠️Here are some tips for efficient data processing in Pandas:

✅ Use efficient data types: Use more memory-efficient data types (e.g. int32 instead of int64, float32 instead of float64) to reduce memory usage.
✅ Load less data: Use the use-cols parameter to load only the columns you need, reducing memory consumption.pd.read_csv()
✅ Chunking: Use the chunksize parameter in to read the dataset in smaller chunks, processing each chunk iteratively.pd.read_csv()
✅ Optimize Pandas dtypes: Use the astype method to convert columns to more memory-efficient types after loading the data, if appropriate.
✅ Parallelize Pandas with Dask: Use Dask, a parallel computing library, to scale Pandas workflows to larger-than-memory datasets by leveraging parallel processing.

🖥Learn more here

Big Data Science

16 Sep 2024 17:59

💡🤖😎10 AI Terms and Aspects That Everyone Needs to Understand and Be Aware of Today
🧐Today, we’ll look at 10 aspects that most broadly cover the field of AI in its various manifestations:
✅ Reasoning/Planning: Modern AI systems can solve problems by using patterns they’ve learned from historical data to understand the information, which is similar to the process of reasoning. The most advanced systems can go further, tackling more complex problems by creating plans and determining a sequence of actions to achieve a goal.
✅ Learning/Inference: There are two stages to creating and using an AI system: learning and inference. Learning can be compared to the process of educating an AI, where it’s given a set of data and it learns to perform tasks or make predictions based on that data.
Inference is the process by which an AI uses learned patterns and parameters to, for example, predict the price of a new home that will soon go on sale.
✅ Small Language Models (SLMs): Compact versions of Large Language Models (LLMs). Both of these types use machine learning techniques to recognize patterns and relationships, allowing them to generate realistic and natural language responses. However, unlike LLMs, which are huge and require a lot of computing power and memory, SLMs like Phi-3 are trained on smaller, curated datasets and have fewer parameters.
✅ Grounded: Generative AI systems can create stories, poems, jokes, and answer research questions. However, they sometimes have difficulty separating fact from fiction or use outdated data, leading to erroneous answers called “hallucinations.” Developers aim to make AI interactions with the real world more accurate through a process called grounding, where the model is connected to current data and specific examples to improve accuracy and produce more relevant results.
✅ Retrieval Augmented Generation (RAG): When developers give AI access to external data sources to make it more accurate and relevant, a technique called Retrieval Augmented Generation (RAG) is used. This approach saves time and resources by adding new knowledge without having to retrain the AI.
✅ Orchestration: AI programs perform many tasks when processing user requests, and an orchestration layer manages their actions in the right order to get the best response. The orchestration layer can also follow the RAG pattern, searching the web for fresh information and adding context.
✅ Memory: Modern AI models technically do not have memory. However, they may have orchestration instructions that help them “remember” information by performing specific steps with each interaction.
✅ Transformers and Diffusion Models: Humans have been training AI systems to understand and generate language for decades, but one of the breakthroughs that has accelerated progress is the Transformer model. Among generative AIs, Transformers are the ones that understand context and nuance the best and fastest.
Diffusion models are typically used to generate images. These models continue to make small adjustments until they create the desired output.
✅ Frontier Models: Frontier models are large-scale systems that push the boundaries of AI and can perform a wide range of tasks with new and advanced capabilities. They are becoming key tools for a variety of industries, including healthcare, finance, scientific research, and education.
✅ GPU: A graphics processing unit is a powerful computing unit. Initially created to improve the graphics in video games, they have now become the real “muscles” of the computing world. And since AI essentially deals with a huge number of computational problems in order to understand language and recognize images or sounds, GPUs are indispensable for AI both at the training stage and when working with finished models.

Big Data Science

09 Sep 2024 17:59

😎3 useful tools for working with SQL tables

SQL Fiddle - A tool for simple testing, debugging and sharing SQL fragments. Add text to the panel, and SQL Fiddle turns it into a script for creating the necessary table. Suitable for both working with databases and practicing SQL skills.

SQL Database Modeler - can create the structure of new tables and relationships between them, connect to existing databases and design changes to them. And all this in a nice graphical interface and with a link to GitHub.

SQLFlow - a simple tool for visualizing SQL queries and displaying dependencies. Allows you to track data lineage and transformations in data when executing queries.

Big Data Science

02 Sep 2024 17:59

⚠️Text2SQL is no longer enough

I recently came across an article in which the authors describe in detail the innovative TAG approach.

Table Augmented Generation (TAG) is a unified general-purpose paradigm for answering natural language questions using databases. The essence of this approach is that we have a model that accepts a natural language query, processes it, and returns a natural language answer.

Thus, Text2SQL only represents the spectrum of interactions between LM and the database. The very essence of these interactions is described using TAG.

📚 Article with a detailed description
🛠 Implementation of the approach

Big Data Science

28 Aug 2024 17:59

🧐What is the difference between DICOM and NIfTI medical image formats

Before we look at the differences between DICOM and NIfTI, let's take a closer look at what each of these formats is individually

🤔What is the DICOM standard?

The DICOM standard — Digital Imaging and Communications in Medicine (DICOM) — is used to exchange images and information, it has been popular for more than a decade. Today, almost every device used in radiology (including CT, MRI, ultrasound and radiography) is equipped with support for the DICOM standard. According to the information from the standard developer (), DICOM allows you to transfer medical images in an environment of devices from different manufacturers and simplify the development and expansion of image archiving and communication systems.

🤔What is the NIfTI standard?

The Neuroimaging Informatics Technology Initiative (NIfTI) was created to work with users and manufacturers of medical devices to address some of the problems and shortcomings of other imaging standards. NIfTI was specifically designed to address these issues in the field of neuroimaging, with a focus on functional magnetic resonance imaging (fMRI). According to the NIfTI definition, the primary mission of NIfTI is to provide coordinated, targeted services, education, and research to accelerate the development and usability of neuroimaging informatics tools. NIfTI consists of two standards, NIfTI-1 and NIfTI-2, the latter being a 64-bit enhancement of the former. It does not replace NIfTI-1, but is used in parallel and supported by a wide range of medical neuroimaging devices and operating systems.

❓What is the difference between DICOM and NIfTI?

1. NIfTI files have less metadata: An NIfTI file does not require as many tags to be filled in as a DICOM image file. There is much less metadata to inspect and analyze, but this is in some ways a disadvantage because DICOM provides users with different layers of image and patient data.

2. DICOM files are often bulkier: DICOM data transfer is governed by strict formatting rules that ensure that the receiving device supports SOP classes and transfer syntaxes, such as the file format and encryption used to transfer the data. When transferring DICOM files, one device talks to another. If one device cannot process the information that the other is trying to send, it will inform the requesting device so that the sender can roll back to a different object (e.g. a previous version) or send the information to a different receiving end. Therefore, NIfTI files are usually easier and faster to process, transfer, read, and write than DICOM image files.

3. DICOM works with 2D layers, while NIfTI can display 3D details: NIfTI files store images and other data in a 3D format. It is specifically designed to overcome the spatial orientation issues of other medical imaging file formats. DICOM image files and associated data are made up of 2D layers. This allows for viewing different sections of an image, which is especially useful when analyzing the human body and different organs. However, with NIfTI, neurosurgeons can quickly identify objects in images in 3D, such as the right and left hemispheres of the brain. This is invaluable when analyzing images of the human brain, which is extremely difficult to evaluate and annotate.

4. DICOM files can store more information: As mentioned above, DICOM files allow medical professionals to store more information in different layers. Structured reports can be created and even images can be frozen so that other clinicians and data scientists can clearly see what the opinion/recommendation is based on.

Big Data Science

14 Aug 2024 17:58

⚠️Attention! Spark = Pandas + Big Data support

Be careful when applying your Pandas knowledge to Spark!!!

Of course, Pandas and Spark operate on the same data type — tables. However, the way they interact with them is significantly different.
For example, the main difference is that Pandas runs in a single process on a single machine and loads all the data into memory, while Spark is designed to work with large distributed data sets and can process terabytes and petabytes of data without loading it entirely into the memory of a single node

However, unfortunately, many programmers often transfer their knowledge from Pandas to Spark, assuming similar architectures, which leads to performance bottlenecks.

You can learn more about solving this problem from this article

Big Data Science

09 Aug 2024 17:59

💡😎The book "PostgreSQL 16 from the inside" is now freely available

The Postgres Professional DBMS developer has released a new book "PostgreSQL 16 from the inside". The electronic version of the textbook is freely available. The author of the book is Egor Rogov, Director of Educational Program Development at Postgres Professional.

The first edition of this textbook, based on version 14 of PostgreSQL, was released in March 2022 and updated to version 15. Due to great reader interest, the company translated the book into English. It later became the most popular thematic publication of 2023 according to Postgres Weekly and was included in the list of professional literature on the official website of the PostgreSQL community.

The current edition of the book "PostgreSQL 16 from the Inside" takes into account readers' comments, corrects typos, and reflects changes that occurred in the PostgreSQL 16 version. Postgres Professional has also updated the localized documentation for PostgreSQL 16.

Big Data Science

05 Aug 2024 14:04

💡😎A startup that revolutionized the way we process data

CRAM is a new memory technology that can reduce energy consumption when processing AI data by 1000 times.

Researchers from the University of Minnesota have developed a new technology, Computational Random-Access Memory (CRAM), that can reduce energy consumption when processing data. Unlike traditional solutions, where data moves between memory and the processor, CRAM allows data to be processed directly in memory cells.

This is achieved through the use of a high-density and reconfigurable spintronic structure embedded in memory cells. Thus, the data does not leave the memory, which minimizes response delays and energy consumption associated with the transfer of information.

With CRAM, data never leaves memory, but is instead processed entirely within the computer’s memory array. This allows a system running an AI computing application to reduce power consumption by “about 1,000 times compared to a state-of-the-art solution,” according to the research team.

Big Data Science

29 Jul 2024 17:59

💡Datasets used to build various ML bases

Iphone dataset - a set of datasets on the basis of which more than 40 thousand dynamic and more than 100 thousand static Gaussians, 20 SE(3) bases were built using Shape of Motion

The training time on 1xGPU A100 using the Adam optimizer with a resolution of 960x720 was just over 2 hours at a rendering speed of 40 frames per second.

According to the results of tests during the training process, Shape of Motion showed good results in the quality and consistency of scene construction.
However, the method still requires optimization for each specific scene and cannot handle significant changes in camera angle. There is also a critical dependence on precise camera parameters and user input to create a moving object mask.

Big Data Science

24 Jul 2024 17:58

⚡️The largest collection of datasets of ~ 1 million pairs of problems and solutions for mathematical competitions

NuminaMath - datasets consisting of 1 million pairs of problems and solutions for various mathematical problems.

🔎Chain of Reasoning (CoT): 860 thousand pairs of problems and solutions created using CoT.

🛠 Tool-Integrated Reasoning (TIR): 73K synthetic solutions derived from GPT-4 with code execution feedback to break complex problems into simpler subproblems that can be solved using Python.

According to the researchers, models trained on NuminaMath achieve best-in-class performance among open-weight models and approach or beat their own models in math competition scores.

Big Data Science

19 Jul 2024 17:59

💡 Large video dataset with long duration and structured annotations

Tencent's MiraData is an off-the-shelf dataset with a total video duration of 16 thousand hours, designed to train text-to-video generation models. It includes long videos (average 72.1 seconds) with high motion intensity and detailed structured annotations (average 318 words per video).

To evaluate the quality of the dataset, a MiraBench benchmark system of 17 metrics assessing temporal consistency, motion in the frame, video quality, and other parameters was even specially created. According to their results, MiroData outperforms other known datasets available in open sources, which mostly consist of short videos with floating quality and short descriptions.

Big Data Science

15 Jul 2024 17:59

🔎Lakehouse architecture: advantages and disadvantages

Lakehouse architecture is designed to provide more flexible and efficient data processing, including data storage, processing and analytics. It is a hybrid approach that combines elements of a traditional Data Warehouse and a Data Lake.

Lakehouse advantages:

1. Data unification: Lakehouse architecture allows you to store structured and unstructured data in one place. This simplifies data access and analysis, eliminating the need for separate systems for each type of data.
2. Cost-effective: By using low-cost data storage solutions such as cloud storage objects, Lakehouse architecture can be more cost-effective compared to traditional data warehouses.
3. Flexibility and Scalability: Lakehouse supports scalability, making it easy to increase data storage and processing power as needed. This is especially important for companies working with large volumes of data and requiring high performance.
4. Compatibility with modern analytical tools: Many modern analytical tools and platforms, such as Apache Spark, Delta Lake and others, integrate with the Lakehouse architecture, providing high performance and reliability of data analysis.

Disadvantages of Lakehouse

1. Implementation Difficulty: Implementing Lakehouse architecture can require significant effort and expense in planning, designing, and configuring the system. This may include training staff and adapting existing processes and tools.
2. Data Quality Management: Merging data from different sources can lead to data quality issues, especially if there are no rigorous data cleaning and validation processes in place.
3. Security and Privacy: Consolidating large amounts of data in one place increases the risks associated with data security and privacy. Additional measures are required to protect data from unauthorized access and leaks.
4. Potential Data Access Latency: In some cases, the Lakehouse architecture may experience latency in data access, especially when processing large volumes of unstructured data.

Thus, Lakehouse architecture offers many benefits such as data unification, cost efficiency and flexibility, making it attractive to many organizations. However, its implementation is associated with certain challenges, including complexity of integration, data quality management and security issues.

Big Data Science

04 Oct 2024 17:59

😎⚡️A powerful dataset generated using Claude Opus.

Synthia-v1.5-I is a dataset of over 20,000 technical questions and answers designed to train large language models (LLM). It includes system prompts styled like Orca to encourage the generation of diverse answers. This dataset can be used to train models to answer technical questions more accurately and comprehensively, improving their performance on a variety of technical and engineering problems.

✅To load the dataset using Python:

from datasets import load_dataset
ds = load_dataset("migtissera/Synthia-v1.5-I")

Big Data Science

30 Sep 2024 17:59

🌎TOP DS-events all over the world in October
Oct 1-2 - AI and Big Data Expo Europe - Amsterdam, Netherlands - https://www.ai-expo.net/europe/
Oct 7-10 - Coalesce - Las Vegas, USA - https://coalesce.getdbt.com/
Oct 9-10 - World Summit AI - Amsterdam, Netherlands - https://worldsummit.ai/
Oct 9-10 - Big Data & AI World - Singapore, Singapore - https://www.bigdataworldasia.com/
Oct 10-11 - COLLIDE 2024: The South's largest data & AI conference - Atlanta, USA - https://datasciconnect.com/events/collide/
Oct 14-17 - Data, AI & Analytics Conference Europe 2024 - London, UK - https://irmuk.co.uk/data-ai-conference-europe-2024/
Oct 16-17 - Spatial Data Science Conference 2024 - New York, USA - https://spatial-data-science-conference.com/2024/newyork
Oct 19 - Oktoberfest - London, UK - https://datasciencefestival.com/event/oktoberfest-2024/
Oct 19 - INFORMS Workshop on Data Science 2024 - Seattle, Washington, USA - https://sites.google.com/view/data-science-2024
Oct 20-25 - TDWI Transform - Orlando, USA - https://tdwi.org/events/conferences/orlando/information/sell-your-boss.aspx
Oct 21-25 - SIAM Conference on Mathematics of Data Science (MDS24) - Atlanta, USA - https://www.siam.org/conferences-events/siam-conferences/mds24/
Oct 23-24 - NDSML Summit 2024 + AI2R Expo - Stockholm, Sweden - https://ndsmlsummit.com/
Oct 28-29 - Cyber Security Summit - San Paulo, Brazil - https://www.cybersecuritysummit.com.br/index.php
Oct 29-31 - ODSC West - California, United States - https://odsc.com/

Big Data Science

25 Sep 2024 17:59

🧐💡A Brief Introduction to MapReduce: Advantages and Disadvantages

MapReduce is a programming model and associated framework for processing large data sets in parallel on distributed computing systems. It includes two main phases: Map (projection) and Reduce (reduction).

Advantages of MapReduce:

✅Scalability: MapReduce easily scales to thousands of machines, allowing it to process huge amounts of data

✅Parallelism: MapReduce automatically distributes tasks across available nodes, executing them in parallel, reducing computational time

✅Fault tolerance: Built-in fault tolerance allows tasks to be restarted in the event of node failure, ensuring completion without data loss

Disadvantages of MapReduce:

✅High I/O Cost: One of the key disadvantages is that data is written and read from disk between the Map and Reduce stages, significantly reducing performance in tasks where fast data transfer is important

✅Lack of interactivity: MapReduce is designed for batch processing, making it inefficient for interactive queries or real-time analysis

✅Shuffle phase requirement: The shuffle phase is often resource intensive and time, making this process a bottleneck in MapReduce performance

✅Low performance for complex tasks: For complex algorithms that require many steps of communication between nodes (e.g. iterative tasks), MapReduce performance degrades

You can also learn more about MapReduce from here

Big Data Science

18 Sep 2024 17:59

💡Creating recommendations for applications with minimal complexity using vector databases

This data not only trains AI systems, but is also the final output that you continue to work with. That's why it's so important to use "good" data. No matter how powerful the model is, if the input is bad data, the output will be the same.

This article is about an example of using the Weaviate database in Streamlit format to simplify working with vector databases. The authors believe that this will allow you to create a powerful search and recommendation system taking into account technical and cost factors.

📚For information, it is worth noting that:
✅Weaviate is an open-source vector database that allows users to store data objects and vector data from machine learning models and easily scales to billions of data objects. .

✅Streamlit is a Python framework. It contains a set of software tools that allow you to transfer a machine learning model to a website. The written "smart" program with this framework can be quickly turned into web applications.

Big Data Science

11 Sep 2024 17:59

🤔Conducting a data quality assessment at Airbnb

✅Airbnb is an online platform for posting and searching for short-term rentals of private housing around the world.

I recently came across an article, where the author describes the process of developing and implementing a data quality assessment methodology, as well as the principles, criteria, and parameters used for this assessment.

As the author notes, the assessment is based on the following principles:

1. Full coverage is an assessment method that can be applied to all data from an entire array, ensuring analysis and processing of information without omissions or limitations. This principle allows for a more complete and accurate study of data, covering the entire set, regardless of its volume or complexity.
2. Automation is a process in which the collection of input data required for the assessment is fully automated, without the need for manual intervention. This principle ensures high speed, accuracy and efficiency in collecting and processing data, which improves the quality of analysis and reduces the time for decision-making.
3. Actionable is a characteristic that means that the data quality assessment is easily accessible and understandable for both producers and consumers of data. This ensures transparency and ease of use of the assessment results, which contributes to more effective interaction and increased trust between all parties.
4. Multidimensionality is a property of the assessment that allows it to be decomposed into various basic components of data quality. This helps to analyze in detail individual aspects affecting the overall quality, such as accuracy, completeness, relevance and consistency, providing a deeper understanding and the ability to target improvement of each component.
5. Evolvability is a characteristic of the assessment, meaning that the criteria and their definitions can adapt and change over time. This flexible approach allows the assessment to remain relevant and effective in the face of changing requirements, new data and technological advances.

Big Data Science

04 Sep 2024 17:59

😎Universal database with embeddings

✅txtai is a universal database of embeddings designed for semantic search, orchestration of large language models (LLM), and management of machine learning workflows. This platform allows you to efficiently process and extract information, use semantic search for text search, and organize and automate tasks related to training and applying machine learning models.

Key features of txtai:

— Includes vector search using SQL, object storage, graph analysis, and multimodal indexing
— Supports embeddings for various data types, including text, documents, audio, images, and videos
— Allows you to build pipelines based on language models to perform various tasks, such as generating suggestions for LLM, answering questions, labeling data, transcription, translation, summarization, and more

🖥 GitHub
🟡 Documentation

Big Data Science

30 Aug 2024 17:59

🌎TOP DS-events all over the world in September
Sep 2-4 - CDAO Melbourne - Melbourne, Australia - https://cdao-mel.coriniumintelligence.com/
Sep 6-7 - Big Data Conference 2024 - Harvard, USA - https://cmsa.fas.harvard.edu/event/bigdata_2024/
Sep 7 - Platzi Conf 2024 - Bogota, Colombia - https://platzi.com/conf/
Sep 9-11 - ECDA 2024 - Sopot, Poland - https://www.ecda2024.pl/
Sep 10-11 - Civo Navigate Europe 2024 - Berlin, Germany - https://www.civo.com/navigate/europe
Sep 11-13 - RTC.ON 2024 - Krakow, Poland - https://rtcon.live/
Sep 18-19 - THE UK’S LEADING DATA, ANALYTICS & AI EVENT - London, UK - https://www.bigdataldn.com/
Sep 23-25 - Data Makers Fest - Alfândega do Porto, Portugal - https://www.datamakersfest.com/
Sep 24 - hayaData 2024 - Tel Aviv, Israel - https://www.haya-data.com/
Sep 24 - APAC Data 2030 Summit 2024 - Singapore, Singapore - https://apac.data2030summit.com/
Sep 24-26 - ICBDE 2024 - London, UK - https://www.icbde.org/
Sep 25-26 - BIG DATA & ANALYTICS MONTRÉAL SUMMIT 2024 - Montreal, Canada - https://bigdatamontreal.ca/
Sep 26-27 - Big Data Conference 2024 - Smaland, Sweden - https://lnu.se/en/meet-linnaeus-university/current/events/2024/conferences/big-data-2024---26-27-sep/
Sep 28-30 - GovAI Summit 2024 - Arlington, United States - https://www.govaisummit.com/
Sep 30-Oct 2 - Ray Summit 2024 - San Francisco, United States - https://raysummit.anyscale.com/flow/anyscale/raysummit2024/landing/page/eventsite

Big Data Science

23 Aug 2024 17:59

⚡️A Scalable Dataset for Tuning Instructions in Software Mathematical Reasoning

The Mathematical Reasoning pipeline emphasizes separating numbers from mathematical problems to synthesize number-independent programs, enabling efficient and flexible scaling while minimizing dependence on specific numerical values.

As the authors note in their paper, experiments in fine-tuning open-source language and code models such as Llama2 and CodeLlama demonstrate the practical benefits of the InfinityMATH dataset.

In addition, these models have shown high reliability on the GSM8K+ and MATH+ benchmarks, which are improved versions of the benchmarks with minor changes to the numerical values.

📊Dataset
📖Research paper

Big Data Science

12 Aug 2024 17:59

⚡️📊OpenAI now provides normal structured JSON with data

I would like to remind you that the JSON mode has been working for about a year, but the outputs of the models corresponded to the declared format in less than half of the cases.

However, there is great news for developers who need good data markup. The updated version gpt-4o-2024-08-06 no longer has this problem: 100% of tests have no errors in the format.

The code and tutorial on using the feature are here.

Big Data Science

07 Aug 2024 17:59

💡😎Interesting Caldera Dataset

The Caldera dataset is an open source scene dataset containing much of the geometry found in the game Call of Duty®: Warzone™.

This includes geometry that can be visualized, as well as some alternate, usually unseen representations used in other calculations. For example, the developers have included volumes here to aid in lighting calculations or simple shapes for collision detection. Excluded are many single-point entities, such as character spawn locations or complex script-based models. As the developers note, they decided not to include textures and materials in this release. That would have added complexity and size to an already heavy scene. They focused on the many connections between spatial elements that can be found in this set, rather than an accurate visual representation.

Big Data Science

31 Jul 2024 18:03

🌎TOP DS-events all over the world in August
Aug 2-4 - MLMI 2024 - Osaka, Japan - https://mlmi.net/
Aug 3-9 - International Joint Conference on Artificial Intelligence (IJCAI) - Jeju, South Korea - https://ijcai24.org/
Aug 5-6 - ICASAM 2024 - Vancouver, Canada - https://waset.org/applied-statistics-analysis-and-modeling-conference-in-august-2024-in-vancouver
Aug 7-8 - CDAO Chicago - Chicago, United States - https://da-metro-chicago.coriniumintelligence.com/
Aug 12-14 - AI4 2024 - Las Vegas, United States - https://ai4.io/vegas/
Aug 16-17 - Machine Learning for Healthcare 2024 - Toronto, Canada - https://www.mlforhc.org/
Aug 19-20 - Artificial Intelligence and Machine Learning - Toronto, Canada - https://www.scitechseries.com/artificial-intelligence-machine
Aug 19-22 - The Bioprocessing Summit - Boston, USA - https://www.bioprocessingsummit.com/
Aug 25-29 - ACM KDD 2024 - Barcelona, Spain - https://kdd2024.kdd.org/
Aug 27 - Azure AI Summer Jam -
Aug 27-29 - ITCN Asia 25th - Karachi, Pakistan - https://itcnasia.com/karachi/
Aug 31 - DATA SATURDAY #52 - Oslo, Norway - https://datasaturdays.com/Event/20240831-datasaturday0052

Big Data Science

26 Jul 2024 17:58

😎💡Benchmark for comprehensive assessment of LLM logical thinking

ZebraLogic is a benchmark based on logic puzzles and is a set of 1000 program-generated tasks of varying difficulty - with a grid from 2x2 to 6x6.

Each puzzle consists of N houses (numbered from left to right) and M features for each house. The task is to determine the unique distribution of feature values across the houses based on the provided clues.
Language models are given one example of the puzzle solution with a detailed explanation of the reasoning process and the answer in JSON format. Models must then solve a new problem, providing both the reasoning progress and the final solution in a given format.

Evaluation Metrics:
1. Puzzle-level accuracy (percentage of completely correctly solved puzzles).
2. Cell-level accuracy (percentage of correctly completed cells in the solution matrix).

🟡 Project Page
🟡 Dataset

Local launch of ZebraLogic as part of the ZeroEval framefork:

# Install via conda

conda create -n zeroeval python=3.10
conda activate zeroeval

# pip install vllm -U # pip install -e vllm 

pip install vllm==0.5.1
pip install -r requirements.txt
# export HF_HOME=/path/to/your/custom/cache_dir/

# Run Meta-Llama-3-8B-Instruct via local, with greedy decoding on `zebra-grid`
bash zero_eval_local.sh -d zebra-grid -m meta-llama/Meta-Llama-3-8B-Instruct -p Meta-Llama-3-8B-Instruct -s 4

Big Data Science

22 Jul 2024 17:58

😎Graph database implemented on the Apache Apache TinkerPop3 framework

HugeGraph is an open-source graph database implemented on the Apache TinkerPop3 framework and fully compatible with the Gremlin query language.

HugeGraph supports the import of over 10 billion vertices and edges and can process queries very quickly (at the ms level).

Typical HugeGraph application scenarios include exploring relationships between objects, association analysis, path finding, feature extraction, data clustering, community detection, and graph construction.

Quick start with Docker:

docker run -itd --name=graph -p 8080:8080 hugegraph/hugegraph
# docker exec -it graph bash

Big Data Science

17 Jul 2024 17:59

⚡️🔎Fully Synthetic Dataset

A huge dataset consisting entirely of synthetic data has appeared on Hugging Face.

The LLM (in this case GPT-4o + VLLM) generates answers by representing itself each time with some character: for example, a chemical scientist or a musician.

Synthetic data can sometimes help a lot (especially when the task is abstract and there is no structured information), but they are still treated with caution. They are not realistic enough, they are not diverse enough, and they potentially harbor hallucinations. It is still unclear whether we will ever be free to use “synthetics”, but it is actively being worked on.

Big Data Science

12 Jul 2024 17:59

💡Another small selection of AI tools for Big Data analytics

KNIME Analytics Platform is a free, open-source platform that allows users to stay at the forefront of data science and has 300+ connectors to various data sources. and integrates with all popular machine learning libraries.

Polymer - artificial intelligence for transforming data into an optimized, flexible and powerful database. All a user needs to do is upload their spreadsheet to the platform to instantly transform it into an optimized database that can then be mined for insights.

IBM Cognos Analytics is a componentized online business intelligence (BI) service that provides access to a wide range of functions for creating business reports, data analysis, event monitoring and metrics to develop effective business decisions.

Akkio is a business intelligence and forecasting tool that allows users to analyze their data and predict potential outcomes. The AI ​​tool allows users to upload their dataset and select the variable they want to predict, which helps Akkio build a neural network around that variable. Like many other tools, Akkio requires no prior programming experience.

Monkeylearn - uses AI data analytics capabilities to help users visualize and reorganize their data. It can also be used to set up text classifiers and text extractors, which help automatically sort data according to topic or intent, and extract product characteristics or user data.