Big Data Science

10 Apr 2024 17:59

💡😎📊Open Source Synthetic Text-to-SQL Dataset
Gretel releases largest open source Text-to-SQL dataset to speed up training of AI models
As of April 2024, the dataset is believed to be the largest and most diverse synthetic text-to-SQL conversion dataset available today, according to the developers.
The dataset contains approximately 23 million tokens, including approximately 12 million SQL tokens, and a wide range of SQL complexity levels, including subqueries, single joins, multiple joins, aggregations, window functions, and set operations.
To load a dataset via the Python API, you need to write the following script:
from datasets import load_dataset
dataset = load_dataset("gretelai/synthetic_text_to_sql")

Big Data Science

05 Apr 2024 17:59

📊😎💡The two largest open datasets for text recognition have been released
Data sets contain millions of real documents, images and texts for text recognition, analysis and document parsing tasks
VQA is a dataset used to develop and evaluate machine learning models capable of answering image-related questions. In the dataset, questions are assigned to each image, as well as the correct answers to these questions. This dataset is supplemented with annotations from Britten's idl_data project. The supplemented dataset can be loaded using a Python script:
from datasets import load_dataset
dataset = load_dataset("pixparse/idl-wds")
PDFA is a set of documents filtered from the SafeDocs corpus, aka CC-MAIN-2021-31-PDF-UNTRUNCATED. This corpus is intended for comprehensive analysis of pdf documents. The supplemented dataset can be loaded using a Python script:
from datasets import load_dataset
dataset = load_dataset("pixparse/pdfa-eng-wds")

Big Data Science

31 Mar 2024 17:59

🌎TOP DS-events all over the world in April
Apr 2-3 - Healthcare NLP Summit - Online - https://www.nlpsummit.org/healthcare-2024/
Apr 9 - Data Architecture New Zealand - Hilton Auckland - https://data-architecture-nz.coriniumintelligence.com/
Apr 9-11 - Google Cloud Next - Las Vegas, US - https://cloud.withgoogle.com/next
Apr 16 - CISO Perth - Perth, Australia - https://ciso-perth.coriniumintelligence.com/
Apr 17-18 - CDAO Germany - https://cdao-germany.coriniumintelligence.com/
Apr 17-18 - AI in Finance – New York, United States - https://ny-ai-finance.re-work.co/
Apr 22-24 - PyCon DE & PyData Berlin 2024 - Berlin, Germany - https://2024.pycon.de/
Apr 22-24 - Machine Learning Prague 2024 - Prague, Czech Republic - https://www.mlprague.com/
Apr 23-24 - The Martech Summit - Singapore - https://themartechsummit.com/singapore
Apr 23-25 - ODSC - Boston, USA - https://odsc.com/boston/schedule-overview/
Apr 24-25 - Gartner Data & Analytics Summit - Mumbai, India - https://www.gartner.com/en/conferences/apac/data-analytics-india
Apr 24-26 - Symposium on Intelligent Data Analysis (IDA) - Stockholm, Sweden - https://ida2024.org/

Big Data Science

27 Mar 2024 16:59

💡📊Training data used in ComCLIP
CLIP (Contrastive Language-Image Pre-Training) is a neural network developed by OpenAI to perform visual as well as language comprehension tasks. The algorithms aim to understand the relationship between text and images.
ComCLIP is an improved version of CLIP for matching text and graphic representations. ComCLIP can mitigate spurious correlations introduced by pre-trained CLIP models and dynamically estimate the importance of each component. Experiments were conducted on four datasets for compositional matching between images and texts.
These datasets are publicly available on the Internet and can be found at this link

Big Data Science

22 Mar 2024 16:59

📚💡Selection of books on various Big Data processing technologies
Spark: The Definitive Guide - the book tells learn how to use, deploy, and maintain Apache Spark with this comprehensive guide, written by the creators of the open-source cluster computing framework code.
Hadoop. Подробное руководство - a book in which it is described thoroughly and clearly You have all the features of Apache Hadoop.
Apache Kafka. Потоковая обработка и анализ данных - the book describes the design principles of the Big Data broker Kafka, reliability guarantees, key APIs and architectural details
Kubernetes в действии - the book talks in detail about Kubernetes - Google's open source software for automating the deployment, scaling and management of applications, scaling and management of Big Data applications
Cassandra: The Definitive Guide: Distributed Data at Web Scale - this guide explains how the Cassandra database management system processes hundreds of terabytes of data while maintaining high availability across multiple processing centers data
MongoDB: полное руководство - This book takes a detailed look at MongoDB, a powerful database management system. Here you can also learn how this secure, high-performance system provides flexible data models, high data availability and horizontal scalability.

Big Data Science

18 Mar 2024 16:58

📊😎💡Selection of services for working with Big Data and integration with various DBMSs
DBeaver is a service that is suitable for integration with various databases, such as MySQL or Oracle. This application is designed for database management. The JDBC interface helps it interact with relational databases. The DBeaver editor allows you to use a large number of additional plugins and gives hints on filling out the code, highlighting the syntax. The application manager supports over 80 databases.
Mixpanel is a system for analytics and analysis of user behavior. It includes features such as:
1. User segmentation
2. Send in-app notifications to your users
3. A/B testing for various notifications
4. Integrating custom surveys into applications via Mixpanel Surveys
App Annie is a service for analytics and obtaining reliable data to make important decisions at all stages of the mobile application business. App Annie will help you study competitors, market conditions, track app downloads, revenue, usage, engagement and advertising. The service also allows you to optimize products for app stores and increase the effectiveness of promotion methods, retention rates and effectively support your target audience. App Annie includes market analytics, multi-store app analytics, and competitor analytics.
Adjust is an optimizer for all product promotion processes. Collects information about where users came to your app page from. It provides a set of measurement and analytics tools that marketers can use to monitor and guide the development of their applications throughout the product lifecycle

Big Data Science

13 Mar 2024 16:59

😎📊Generic set of annotated images
The ImageNet dataset includes 14,197,122 annotated images structured according to the WordNet hierarchy.
Since early 2010, this dataset has been used in the ImageNet Large Scale Visual Recognition Challenge (ILSVRC) and serves as a standard for image classification and object detection tasks.
This large public dataset contains images that have been manually annotated for training purposes.

Big Data Science

06 Mar 2024 16:59

📊💡OAC: advantages and disadvantages
Oracle Analytics Cloud (OAC) is a powerful data analytics tool that delivers business intelligence capabilities in the cloud.
Benefits of Oracle Analytics Cloud:
1. Extensive data analysis capabilities: OAC provides a wide range of tools for data visualization, reporting and trend analysis. It integrates data from various sources, providing a comprehensive view of business processes.
2. Use of cloud technologies: Oracle Analytics Cloud is built on cloud technologies, which provides scalability and flexibility in processing large volumes of data. This also reduces the burden on the company's internal IT resources.
3. Integration with other Oracle products: OAC integrates well with other Oracle products such as Oracle Database, Oracle Cloud Infrastructure and others. This provides a single workspace for data and ensures compatibility with existing systems.
4. Data Security: Oracle Analytics Cloud provides a high level of data security, including encryption mechanisms and access control.
5. Automated Analysis and Machine Learning: OAC provides automated data analysis and machine learning integration capabilities that enable companies to identify hidden trends and predict future events.
Disadvantages of Oracle Analytics Cloud:
1. Implementation Difficulty: Deploying Oracle Analytics Cloud can be a complex process that requires specific technical skills. This can be challenging for smaller companies or organizations with limited resources.
2. Cost of Use: Paid OAC licenses and maintenance can be expensive for small businesses. It is necessary to carefully evaluate budgetary options before deciding to use this platform.
3. Limited UI Flexibility: Despite its extensive capabilities, OAC's user interface may be less flexible than some competitors, which can make it difficult to tailor to specific business needs.
Overall, Oracle Analytics Cloud is a powerful analytics solution, but companies must carefully weigh its advantages and disadvantages based on their business goals and technical capabilities.

Big Data Science

29 Feb 2024 16:59

🌎TOP DS-events all over the world in March
Mar 3-5 - Big Data Minds 2024 - Berlin, Germany - https://www.big-data-minds.eu/
Mar 3-5 - Annual Conference of the Association for Clinical Data Management - Copenhagen, Denmark - https://acdmconference.org/
Mar 6 - Admin & Data Forum 2024 - London, UK - https://event.professionalpensions.com/adminanddataforum/en/page/home
Mar 6-7 - Big Data & AI World - London, UK - https://www.bigdataworld.com/
Mar 13 - Data & Analytics in Healthcare 2024 - Melbourne, Australia - https://datainhealthcare.coriniumintelligence.com/
Mar 14 - Data Management Summit London - London, UK - https://a-teaminsight.com/events/data-management-summit-london/
Mar 17-21 - NVIDIA GTC 2024 - San Juse, USA - https://www.nvidia.com/gtc/
Mar 19-22 - KubeCon + CloudNativeCon - Paris, France - https://events.linuxfoundation.org/kubecon-cloudnativecon-europe/
Mar 26-28 - Microsoft Data & AI Conference 2024 - Las Vegas, USA - https://azuredataconf.com/#!/
Mar 28 - Data and AI Summit 2024 - Richmond, USA - https://rvatech.com/rvatech-events/2024-rvatech-data-summit/

Big Data Science

22 Feb 2024 16:59

📊💡Dataset for setting up mathematical models
OpenMathInstruct-1 is a fresh synthetic dataset from NVIDIA, created for training mathematical models. This dataset includes 1.8 million problem-solution pairs intended for training.
As the developers note, this dataset was created by synthesizing code interpreter solutions for GSM8K and MATH, two popular calculus tests, using the recently released and permissively licensed Mixtral model.

Big Data Science

11 Feb 2024 16:59

💡Data storage vs. Data Lake: advantages and disadvantages
Data warehouses and Data Lakes are two different approaches to managing and storing data in an organization. Let's consider the main aspects of each of them.
Data store:
Advantages:
1. Structured Data: Data warehouses are usually designed to store structured data, making it easier to analyze and process.
2. Performance: Data warehouses use optimized structures to access data quickly, resulting in high query performance.
3. Ready to use: The data in the warehouse is pre-processed and organized, making it ready for use for business intelligence and reporting.
Flaws:
1. Limited data types: Data warehouses can be less flexible when dealing with diverse data types, such as unstructured or semi-structured data.
2. Difficulty scaling: As the volume of data increases, storing and processing it in a warehouse can become more complex and require additional resources.
Data Lake:
Advantages:
1. Flexibility in data types: Data Lake provides the ability to store unstructured and semi-structured data, making it suitable for a variety of data.
2. Scalability: Data Lake easily scales with the growth of data volume, providing increased performance and storage of large volumes of information.
3. On-the-fly data processing: The ability to analyze data in real time allows you to quickly use information for decision making.
Flaws:
1. Management Complexity: Managing a Data Lake may require more complex processes and strategies to avoid clutter and maintain data quality.
2. Unoptimized access: Since data in a Data Lake is stored in its original form, accessing it may require additional effort to optimize queries.
Thus, the choice between a data warehouse and a Data Lake depends on the unique needs of the business and the nature of the data. In some cases, the optimal solution may be a combination of both approaches, providing a comprehensive approach to data management in an organization.

Big Data Science

04 Feb 2024 16:59

🧐💡Firebird DBMS: advantages and disadvantages
Firebird is an open relational database with high performance and advanced capabilities.
Advantages:
1. Open Source: Firebird is distributed under an open source license (InterBase Public License). This allows users to freely use, modify and distribute the software without restrictions.
2. Multi-user support: Firebird provides efficient multi-user functionality, making it suitable for deployment in large enterprise environments.
3. Transactional security: Firebird supports ACID properties (atomicity, consistency, isolation, durability) to ensure transactional data integrity.
4. Multi-tier transaction architecture: Firebird uses a multi-tier transaction architecture, which allows multiple transactions to be executed simultaneously and prevents data locks.
5. SQL standard support: Firebird complies with SQL standards and has advanced features such as support for nested transactions and triggers.
Flaws:
1. Limited ecosystem and tools: Firebird may have a more limited ecosystem and tools compared to more common DBMSs such as MySQL, PostgreSQL or Microsoft SQL Server.
2. Limited GUI support: Firebird may not have as advanced database management tools as some competitors.
3. Limited Community: Compared to some other database management systems, Firebird may have a smaller community of users and developers, which may affect the availability of support and resources for developers.
In general, the choice of DBMS depends on the specific requirements of the project, and Firebird may be a good option for certain use cases, especially when openness and reliability are important.

Big Data Science

26 Jan 2024 16:59

😎💡Little-known but very useful DBMS
TimescaleDB takes PostgreSQL functionality and adds time series to it! Created as an extension to PostgreSQL, this database comes into its own when you deal with large-scale data that changes over time - such as data from IoT devices
FaunaDB is an online distributed transaction processing database with ACID properties. Due to this, high data processing speed and reliability are achieved. FaunaDB is based on technology pioneered by Twitter and was created as a startup by members of the social network's development team.
KeyDB is a Redis fork developed by a Canadian company and distributed under the free BSD license. There is support for multithreading
Riak (KV) is a distributed NoSQL key-value database. Riak CS is designed to provide simplicity, availability, distribution of cloud storage of any scale, and can be used to build cloud architectures - both public and private - or as infrastructure storage for highly loaded applications and services.
InfluxData is designed to monitor metrics and events in the infrastructure. The main focus is storing large amounts of time-stamped data (such as monitoring data, application metrics, and sensor readings) and processing them under high write load conditions.

Big Data Science

22 Jan 2024 16:59

💡📊Selection of libraries for data analysis
Lux is an add-on to the popular Pandas data analysis package. It allows you to quickly create visual representations of data sets and apply basic statistical analysis with a minimum amount of code.
Pandas-profiling - helps generate a profiling report. This report gives a detailed overview of the variables in your dataset. It provides insight into statistics for individual characteristics of the data, such as the distribution, as well as the mean, minimum and maximum values. The same report provides insight into correlations and interactions between variables.
Sweet-Viz - provides fast visualization and analysis of data. Sweet-Viz's main selling point is its extensive HTML dashboard with useful views and data summaries, which is generated by executing just one line of code.
D-Tale is a Python library that provides an interactive and user-friendly interface for visualizing and analyzing Pandas data structures. It uses Flask as the backend and React as the frontend, making it easy to view and explore Pandas data frames, Series objects, MultiIndex, DatetimeIndex and RangeIndex. It integrates easily with Jupyter, Python terminals and ipython.
AutoViz is a Python library that provides automatic data visualization capabilities, allowing users to visualize data sets of any size with just one line of code. The program automatically generates reports in various formats, including HTML and Bokeh, and allows users to interact with the generated HTML reports.
KLib is a Python library that provides automatic exploratory data analysis (EDA) and data profiling capabilities. It offers various features and visualizations to quickly explore and analyze data sets.
SpeedML is a Python library that aims to speed up the development process of a machine learning pipeline. It integrates commonly used ML packages such as Pandas, NumPy, Scikit-learn, XGBoost and Matplotlib. SpeedML also provides functionality for automated EDA

Big Data Science

14 Jan 2024 16:59

💡😎Databricks Lakehouse: advantages and disadvantages
Databricks Lakehouse is a concept that combines the functionality of a data lake and a data warehouse to provide more efficient data management.
Benefits of Databricks Lakehouse:
1. Single space for data storage: Lakehouse provides a single storage for data, combining the advantages of a data lake (flexibility, scalability) and a data warehouse (structured queries optimized for analytics).
2. Scalability: Databricks Lakehouse allows you to efficiently scale data storage and processing, supporting large volumes of information.
3. Support for structured and unstructured data: Lakehouse provides the ability to store and process both structured and unstructured data, making it versatile for various types of information.
4. Using Apache Spark: Databrix includes Apache Spark, which provides high performance and supports big data processing.
Disadvantages of Databricks Lakehouse:
1. Implementation Difficulty: Implementing and configuring Databricks Lakehouse can be challenging, especially for organizations that have not previously worked with similar technologies.
2. Dependency on cloud solutions: For many companies, using Databricks Lakehouse may imply dependence on cloud services, which may cause certain limitations.
3. Cost: Using Databricks Lakehouse, especially in the cloud, can come with additional costs, making it less affordable for smaller businesses.
4. Necessity of data preparation: Working effectively with Lakehouse often requires preliminary data preparation, which may require additional effort.
5. Data management complexity: Managing data in a single space can be a challenge, especially when dealing with large volumes of information and different types of data.

Big Data Science

08 Apr 2024 17:59

📊📉Selection of Python libraries for working with spatial data
Earth Engine API - allows you to access Google Earth Engine's vast collection of geospatial data and perform analysis tasks using Python.
TorchGeo (PyTorch) - Provides tools and utilities for working with geospatial data in PyTorch.
Arcpy (Esri) is a Python library provided by Esri for working with geospatial data on ArcGIS platform. It allows you to automate geoprocessing tasks and perform spatial analysis.
Rasterio is a library for reading and writing geospatial raster datasets. It provides efficient access to raster data and allows you to perform various operations with geodata.
GDAL (Open-Source Geospatial Foundation) is a powerful library for reading, writing and manipulating geospatial raster and vector data formats.
Shapely is a library for geometric operations in Python. It allows you to create, manipulate and analyze geometric objects.
RSGISLib - has functions for processing thermal images, including radiometric correction, earth surface temperature estimation.
WhiteboxTools is a library for geospatial analysis and data processing. It offers a complete set of tools for tasks such as terrain analysis, hydrological modeling and LiDAR data processing.

Big Data Science

03 Apr 2024 17:59

⚔️Relational DBMS vs NOSQL DBMS: advantages and disadvantages
Database implementation is a fundamental element of modern information technology. In the world of databases, there are two main paradigms: relational DBMS and NoSQL DBMS. Each of them has its own advantages and disadvantages, which should be taken into account when choosing the right one for a particular task.
Relational databases are based on a data model known as the relational model. In such databases, data is stored in the form of tables, which consist of rows (records) and columns (fields). The data structure is defined by a predefined schema that describes the data types of each column.
Advantages of relational DBMS:
1. Data structure: Relational DBMS stores data in the form of tables, which makes it easy to understand and organized.
2. ACID properties: Guarantees atomicity, consistency, isolation and durability of transactions, making them reliable for applications that require a high degree of data integrity.
3. SQL Language: A powerful and widely used query language that provides standardization and ease of working with data.
Flaws:
1. Vertical scaling: Relational DBMSs can face vertical scaling limitations, which means that when they reach their performance limits they will have to be migrated to more powerful, and often more expensive, servers.
2. Schema Complexity: Changing the data schema can be difficult and require additional effort and time.
3. Difficulty of horizontal scaling: Even with data partitioning techniques, horizontal scaling of relational DBMSs can be complex and require additional configuration and optimization work.
NoSQL databases are designed to work with unstructured and semi-structured data. They offer a flexible data schema, which allows you to store data without explicitly defining the schema in advance.
Advantages of NOSQL:
1. Flexibility of data structure: NoSQL DBMSs allow you to store unstructured data, making them an ideal choice for applications with changing data requirements.
2. Horizontal scalability: Many NoSQL databases are designed to scale horizontally, making them suitable for handling large amounts of data and high workloads.
Flaws:
1. Lack of ACID properties: Unlike relational DBMSs, NoSQL databases can sacrifice some ACID properties in favor of performance and scalability.
2. Limited support for SQL query language: Some NoSQL DBMSs may have limited query language functionality, which can make it difficult to perform complex queries or analytical operations.
The choice between relational and NoSQL DBMS depends on the specific requirements and characteristics of the project. Relational DBMSs provide high data integrity, while NoSQL DBMSs allow you to work with large volumes of unstructured data and provide flexibility and scalability.

Big Data Science

29 Mar 2024 16:59

💡😎💊Google published a new dataset of skin condition images
SCIN is an open access dataset that contains data on skin condition. This dataset was collected from volunteer Google Search users in the United States using a special application.
SCIN contains 10,000+ images for common dermatological diseases. Materials include images, medical history and symptom information, and self-reported Fitzpatrick skin type.

📄Documentation

☁️Download link

Big Data Science

25 Mar 2024 16:58

💡😎Datasets for the task of converting text to sound
FAIR has released a project for a system for converting text into sound into the public domain.
In addition to the main project, there are also datasets in JSON format.
Detailed instructions for using datasets can be found here

Big Data Science

20 Mar 2024 16:59

⚔️😎💡ClickHouse vs Greenplum
Clickhouse and GreenPlum are well-known DBMSs for big data analysis that are very popular. However, there are criteria by which it is necessary to unambiguously choose which of the DBMS data to use in a given situation. To do this, let's look at their main advantages and disadvantages.
Advantages of ClickHouse:
1. High performance: ClickHouse is designed for analytical tasks and has a high speed of executing requests for reading large amounts of data. This makes it an ideal choice for data analytics and OLAP (Online Analytical Processing)
2. Efficient data compression: ClickHouse uses various data compression methods, which can significantly reduce the amount of stored information without loss of performance.
3. Horizontal scaling: ClickHouse easily scales horizontally, which allows you to increase system performance by adding new nodes.
Disadvantages of ClickHouse:
1. Limited transaction support: ClickHouse is mainly focused on analytical tasks and does not have full transaction support, which can be a problem for some applications.
2. Limited feature set: Despite its performance, ClickHouse may not be sufficient for some complex analytical tasks due to the limited set of built-in features.
Greenplum benefits:
1. Transaction Support: Greenplum provides full support for transactions and ACID (Atomicity, Consistency, Isolation, Durability), making it an ideal choice for OLTP (Online Transactional Processing) and OLAP applications.
2. Wide Range of Features: Greenplum offers a rich set of built-in features and analytical processing capabilities, making it suitable for various types of analytical tasks.
3. Support for distributed transactions: Greenplum provides support for distributed transactions and scales horizontally to handle large volumes of data.
Disadvantages of Greenplum:
1. Complexity to manage: Greenplum may require more effort and experience to manage and configure, especially when dealing with large clusters.
2. Less efficient data compression: Compared to ClickHouse, Greenplum may not provide the same high level of data compression, which may result in higher disk space usage and lower performance
Ultimately, the choice between ClickHouse and Greenplum depends on the specific needs of the task. ClickHouse is better suited for analytical workloads with high performance requirements, while Greenplum may be the preferred choice for applications where transaction support and a wide range of features are important.

Big Data Science

15 Mar 2024 16:59

💡⚔️Sensei will tell you
Sensei is a relatively new Python tool for generating synthetic data using systems such as OpenAI, MistralAI or AnthropicAII.
To start, you need to make the following preset:
pip install openai mistralai numpy
The developers also wrote detailed instructions for setup.

Big Data Science

10 Mar 2024 16:58

🌲💡New dataset about forests
FinnWoodlands is a dataset that includes 4226 manually annotated features, of which 2562 features (60.6%) correspond to tree trunks classified into three different instance categories, and namely "Spruce", "Birch" and "Pine".
In addition to tree trunks, there are object annotations "Obstacles", as well as semantic classes "Lake", "Earth" and "Path".
This dataset can be used in various applications where a holistic view of the environment is important. It provides an initial benchmark using three models for instance segmentation, panoptic segmentation, and depth filling.
Overall, FinnWoodlands consists of stereo RGB images, point clouds and sparse depth maps, as well as reference annotations for semantic segmentation.

Big Data Science

05 Mar 2024 16:59

📊💡DeltaLake: advantages and disadvantages
Delta Lake is an abstraction layer for working with data in data warehouses. Delta Lake provides additional capabilities and data integrity guarantees for storing and processing large volumes of data.
Delta Lake benefits:
1. Transactional Consistency: Delta Lake provides ACID transactions, ensuring transactional data consistency. This ensures reliable operations and data integrity management.
2. Partitioning: Delta Lake supports data partitioning, which improves query performance and data management. Partitioning allows you to effectively filter data based on certain criteria.
3. Improved Performance: Delta Lake optimizes queries and operations on data, leading to improved performance compared to conventional data warehouses.
4. Streaming Data Processing: Delta Lake supports streaming data processing, allowing you to instantly update and analyze data in real time.
Disadvantages of Delta Lake:
1. Difficulty in Setup: Some users may find it difficult to set up and use Delta Lake due to its advanced functionality.
2. Compatibility: Compatibility issues may arise when integrating Delta Lake with other tools and storage systems.
Overall, Delta Lake provides powerful tools for data management and processing, but its use should be considered based on the specific project requirements and team experience.

Big Data Science

24 Feb 2024 16:59

💡📊😎Dataset for virtual reality
Meta announced new projects in the field of artificial intelligence (AI) and an update to its Ego-Exo initiative, aimed at solving problems associated with technologies focused on providing a first-person perspective.
The company released a dataset called Ego-Exo4D. As the developers note, the project will help high-quality training of AI models with complex human skills and will be suitable for creating applications for virtual reality systems, robotics, and much more.
Ego-Exo4D contains three, carefully synchronized natural language datasets combined with video and expert commentary, including more than 1,400 hours of video, as well as benchmark annotations.

Big Data Science

16 Feb 2024 16:59

💡📊Startup for communication with databases
The team of the groql startup from Novosibirsk, the winner of the autumn session of A:START 2023, has developed an application that allows the user to communicate with databases in natural (Russian) language without programming experience and receive visualizations in the form of graphs, charts and graphs. Another advantage of the program is that it works on the basis of AI. Groql helps you translate a query from natural language to SQL. The user can describe the features of the databases and the AI will take them into account when working.
The main advantage of this startup is its visual presentation of data. After processing the request, the user will see a graphical representation of the data, which will help to better understand the relationships between various data. As the developers note, this can help the employer reduce costs by reducing time and simplifying work with data.
Read more: https://habr.com/ru/articles/791358/

Big Data Science

07 Feb 2024 16:59

💡Airbyte: advantages and disadvantages
Airbyte is an open data integration platform designed to simplify the data capture, transformation, and transfer (ETL) process. It is designed to help companies easily share data between different sources and purposes.
Airbyte advantages:
1. Open Source: Airbyte provides open source code which allows users to modify and customize the platform as per their requirements.
2. Ease of Use: Airbyte's interface is user-friendly and intuitive. Users can create and manage connectors for various data sources without the need for extensive technical knowledge.
3. Scalability: The platform provides a scalable architecture, making it suitable for processing large volumes of data.
4. Supports a large number of connectors: Airbyte comes with many built-in connectors for popular data sources such as databases, APIs, cloud services and others.
5. GUI and versioning: Visual tools and versioning make it easy to create, track, and manage your integration configurations.
Flaws:
1. Missing some connectors: Despite the wide range of supported data sources, there may be situations where the required connector is missing.
2. Does not support real-time: Airbyte does not currently provide full real-time support for all data sources.
Overall, Airbyte is a promising data integration tool that can be useful in cases where ease of use, openness, and scalability are important.

Big Data Science

31 Jan 2024 16:59

🌎TOP DS-events all over the world in February
Feb 1-2 - Cloud Technology Townhall Tallinn 2024 - Tallinn, Estonia - https://cloudtechtallinn.com/
Feb 2 - Beyond Big Data: AI/Machine Learning Summit 2024 - Pittsburgh, USA - https://www.pghtech.org/events/BeyondBigData2024
Feb 2 - Nordic AI & Metaverse Summit - Copenhagen, Denmark - https://www.danskindustri.dk/arrangementer/soeg/arrangementer/salg-og-marketing/nordic-ai--metaverse-summit-2024/
Feb 2-3 - National Big Data Health Science Conference 2024 - Columbia, USA - https://www.sc-bdhs-conference.org/
Feb 2-5 - International Conference on Big Data Management 2024 - Zhuhai, China - https://www.icbdm.org/
Feb 6 - TINtech London Market 2024 - London, UK - https://www.the-insurance-network.co.uk/conferences/tintech-london-market
Feb 6 - Big Data III and Artificial Intelligence 2024 - London, UK - https://www.soci.org/events/fine-chemicals-group/2024/big-data-iii-and-artificial-intelligence
Feb 5-7 - IEEE International Conference On Semantic Computing 2024 - California, USA - https://www.ieee-icsc.org/
Feb 11-14 - Summit For Clinical Ops Executives 2024 - Orlando, USA - https://www.scopesummit.com/
Feb 22-23 - 9TH WORLD MACHINE LEARNING SUMMIT - Bangalore, India - https://1point21gws.com/machine-learning/bangalore/

Big Data Science

25 Jan 2024 16:59

😎💡📊In search of the hidden: little-known Python libraries for data analysts
PyCaret - An automated machine learning library that simplifies the transition from data preparation to modeling. PyCaret includes features for automatic model comparison, data preprocessing, and integration with MLflow for easy experimentation.
Vaex - A library for lazy loading and efficient processing of very large data. Great for analyzing large datasets with limited computing resources. aex allows you to efficiently work with datasets containing billions of rows, minimizing memory usage and optimizing performance.
Streamlit - A tool for quickly creating interactive web applications for data analytics. Streamlit can be used to develop applications that demonstrate machine learning results, such as image classification or time series forecasting.
Dask - Designed for parallel computing and working with large datasets. Ideal for scaling analytical operations and processing large volumes of data. Dask provides compatibility with tools like Pandas and Numpy and allows you to perform complex calculations on clusters.
Dash by Plotly - Framework for creating analytical web applications. Ideal for creating interactive dashboards and complex data visualizations. Dash allows you to create rich web applications for data analysis, such as visualizing company financial performance or market data trends.

Big Data Science

18 Jan 2024 16:59

💡📉Dataset programming is no longer a problem
Snorkel - a framework for data programming. The approach of this framework is to use various heuristics and a priori knowledge to automatically label datasets. The project started at Stanford as a tool to help mark up datasets for the information extraction task, and now the developers are creating a platform for use by external customers.
Snorkel's arsenal includes three key tools:
-marking functions for creating a dataset;
-transforming functions for dataset augmentation;
-slicing functions that highlight subsets in the dataset that are critical for the performance of learning models.

Big Data Science

10 Jan 2024 16:59

📚A selection of books for immersion in the world of time series analysis
Time series analysis and forecasting - considered time series indicators, main types of trends and methods for their recognition, methods for estimating fluctuation parameters, measuring the stability of series levels and dynamic trends, modeling and time series forecasting. Designed for persons with knowledge of the general theory of statistics.
Practical analysis of time series: forecasting with statistics and machine learning - modern technologies for analyzing time series data are described here and examples of their practical use in a variety of subject areas are given. It is designed to help solve the most common problems in the study and processing of time series using traditional statistical methods and the most popular machine learning models.
Elementary theory of analysis and statistical modeling of time series - the book contains the theoretical and probabilistic foundations of the analysis of the simplest time series, as well as methods and techniques for their statistical modeling (simulation ). The material on elementary probability theory and mathematical statistics is presented briefly using the analogy of probabilistic schemes and supplemented with results on the theory of series and criteria of randomness.
Statistical analysis of time series - monograph by a famous American specialist in mathematical statistics contains a detailed presentation of the theory of statistical inference for various probabilistic models. Methods for representing time series, estimating the parameters of corresponding probabilistic models, and testing hypotheses regarding their structure are outlined. The extensive material collected by the author, previously scattered across various sources, makes the book a valuable guide and reference book.
Time series. Data processing and theory () - the monograph is devoted to the study of times series found in various fields of physics, mechanics, astronomy, technology, economics, biology, medicine. The main orientation of the book is practical: methods of theoretical analysis are illustrated with detailed examples, and the results are clearly presented in numerous graphs.