1443
All ngrams about Natural Language Processing that are of interest to @iamdddaryna
TextDetox is back
🚀 TextDetox @ CLEF 2025 – A Multilingual Challenge for Textual Style Transfer! 🚀
Toxicity in online conversations is a growing concern, but NLP can help make a difference! Our Text Detoxification Shared Task invites researchers and practitioners to develop innovative methods for transforming toxic language into neutral text—while preserving the original meaning and changing the author’s original style minimally.
What is this sh*t all about? -> What is this all about?
🌍 Why join?
✅ Work with 15 diverse languages - find your language! English (en), Spanish (es), German (de), Chinese (zh), Arabic (ar), Hindi (hi), Ukrainian (uk), Russian (ru), Amharic (am), Italian (it), Tatar (tt), Japanese (ja), Hebrew (he), French (fr) and Hinglish (hin).
✅ Tackle multilingual & cross-lingual detoxification – train on 9 languages, generalize to 6 new ones!
❔Can LLMs tackle text style transfer for so many languages – let’s find out!
✅ Get featured on our final leaderboard with advanced evaluation metrics.
✅ Publish & present your work at CLEF 2025 in sunny Madrid, Spain! ☀️
🔗 How to participate?
📢 Stay updated via our Google Group https://groups.google.com/g/textdetox-clef2025
📝 Register officially – Sign up for the CLEF conference https://clef2025.clef-initiative.eu/index.php?page=Pages/registration.html
💡 More details & datasets on our website https://pan.webis.de/clef25/pan25-web/text-detoxification.html
🤗 Everything is opensourced: https://huggingface.co/textdetox
📅 Key Dates:
🔹 Registration is open now! Join our google group: https://groups.google.com/g/textdetox-clef2025 and stay tuned for updates!
🔹 April 25, 2025 – Registration closes
🔹 May 1, 2025 – Test phase starts
🔹 May 10, 2025 – Evaluation ends
🔹 May 30, 2025 – Paper submission
🔹 September 9-12, 2025 – Present at CLEF 2025 in Madrid!
Join us in shaping responsible and multilingual AI for a safer digital world! 🌏✨
Crafting Tomorrow’s Headlines: Neural News Generation and Detection in English, Turkish, Hungarian, and Persian
We present a new benchmark for ai-generated texts, specifically, news, detection🔍
Four languages: English, Turkish, Hungarian, and Persian
Various LLMs: BloomZ, LLaMa, Mistral, Mixtral, and GPT-4
🤗https://huggingface.co/datasets/tum-nlp/neural-news-benchmark
Can your train such a classifier that will be able to detect GPT-4 texts?🤔 Or, maybe, other LLMs can detect ai-generated texts?
Find out in our new preprint:
📜https://arxiv.org/abs/2408.10724
NLP for Positive Impact Workshop
We are thrilled to invite submissions to the Third Workshop on NLP for Positive Impact!
🔗 Workshop Website: https://sites.google.com/view/nlp4positiveimpact
📅 Important Dates:
Submission Deadline: June 15, 2024, 11:59 PM AoE
Commitment Deadline: August 20, 2024
Notification of Acceptance: September 20, 2024
Camera-Ready Papers Due: October 3, 2024
Workshop Date: Co-located with EMNLP 2024 in November, Miami
This workshop is a platform to explore how all skyrocketing NLP 🚀 can address critical global issues and support the UN sustainability goals 🌍 We are looking for innovative research that focuses on the societal impact of NLP, including areas like healthcare, education, inequality, climate change, and more.
🌟 Special Theme: Tackling digital violence through NLP and AI 🌟
We encourage interdisciplinary collaborations and value submissions that connect NLP with other fields and NGOs. Submissions should include a discussion on the ethical and societal implications of the work, aiming for a positive impact.
📜 Submission Types:
Case studies of real-world deployments
Position papers proposing new tasks or directions
Literature reviews
Philosophical discussions
Approaches to interdisciplinary collaboration
Ethical considerations
Join us in Miami and share your research with a vibrant community dedicated to using NLP for the greater good. Let's harness the power of language-oriented AI to make a positive difference in the world!
📧 Contact: nlp4pi.workshop@gmail.com
Looking forward to your contributions!
Organizers:
Zhijing Jin (Max Planck Institute & ETH Zurich)
Daryna Dementieva (Technical University of Munich)
Steven Wilson (Oakland University)
Oana Ignat (University of Michigan)
Jieyu Zhao (University of Maryland, College Park)
Joel Tetreault (Dataminr, Inc.)
Rada Michaela (University of Michigan)
TextDetox CLEF 2024: Final week of the dev phase
We would like to remind you that this week is a final week of the dev phase of our multilingual TextDetox shared task:
https://pan.webis.de/clef24/pan24-web/text-detoxification.html
🤗https://huggingface.co/textdetox
On April, 22nd, the official registration to CLEF-2024 will be closed, so, please, register here if you have not done this yet:
https://clef2024-labs-registration.dei.unipd.it/
Also, we would like to remind you that still the dev phase leaderboard is open and you are welcomed to make your submission!
Please, submit to Codalab:
https://codalab.lisn.upsaclay.fr/competitions/18243
or to TIRA (as an additional option in case of technical problems):
https://www.tira.io/task/pan24-text-detoxification
Otherwise, stay tuned for the test set release!
Has It All Been Solved? Open NLP Research Questions Not Solved by Large Language Models
PhD application season is starting. If you were afraid, that the only topic you will be suggested is only to prompt LLMs, here are good scientifically proved news for you — there are still plenty to do in NLP!
Amazing colleagues from the Michigan University has prepared a list of still open NLP research questions, 45 of them! Including:
* Multilinguality
* Reasoning
* Knowledge Bases
* Language Grounding
* Computational Social Science
* Online Environments
* Child Language Acquisition
* Non-verbal Communication
* Synthetic Datasets
* Interpretability
* Efficient NLP
* NLP in Education
* NLP in Healthcare
* NLP and Ethics
Yes, in some direction, we have gone already a long way, so other topics are becoming important and just possible already to explore✨
Check the full text (is appearing at COLING):
https://arxiv.org/abs/2305.12544
P.S. And I am reminding, that we are having multilingual safe-language important shared task on texts detoxification — start you first research experiments now😉
Ukrainian Texts Classification Corpora p2
We continue to enrich datasets for the classification of texts in the Ukrainian language. This time, we worked on the translation of English-language data into Ukrainian and obtained:
1. Ukrainian NLI corpus: https://huggingface.co/datasets/ukr-detect/ukr-nli-dataset-translated-stanford translated from Stanford SNLI.
2. Ukrainian Formality corpus: https://huggingface.co/datasets/ukr-detect/ukr-formality-dataset-translated-gyafc translated from English GYAFC
3. In addition to the toxicity corpus presented previously, translated data from the English Jigsaw Toxicity Classification dataset https://huggingface.co/datasets/ukr-detect/ukr-toxicity-dataset-translated-jigsaw
You are very welcomed to use and test them😉
ELLIS Winter School on Foundation Models
Amsterdam 12-15th March
https://amsterdam-fomo.github.io/
Foundation Models, and their origin, analysis and development have been typically associated with the US and Big Tech. Yet, a critical share of important insights and novel approaches do come from Europe, both within academia and industry. Part of this winter school's goal is to highlight these fresh perspectives and give the students an in-depth look into how Europe is guiding its own research agenda with unique directions and bringing together the community. The workshop will take place at the University of Amsterdam.
Lectures from top researchers from DeepMind, Google Research, and top EU unis.
Deadline to apply: 15th February 2024 23:59 CET
Happy New Year 2024
Thank you for being interested in NLP and my view on it 🤩
For new year, I have some new ideas for the community -- stay tuned 😉
Be professional, believe in yourself, be open for new ideas, and all other positive tokens in your texts 🥳
My PyData&Conf Berlin 2023: Texts Detoxification
It was a pleasure for me to be part of PyData&Conf Berlin 2023 — amazing scientist and developers all over Europe come together to discuss and share experience in cutting edge data science. Of course, there were a lot of talks about LLMs 😉
Firstly, I want to invite you to take a look about my research in texts detoxification. Even with all advances, our models are still actual in the field of toxic speech combating: [video]
Secondly, other I recommend to pay attention to other talks that I personally found interesting:
* Keynote talk: Miroslav Šedivý: Lorem ipsum dolor sit amet. A lot of fun facts about different European languages 😃
* Erin Mikail Staples, Nikolai: Improving Machine Learning from Human Feedback. A lot of attention to HF right now, showcase of a library to help you with it.
* Ines Montani: Incorporating GPT-3 into practical NLP workflows. Told you, a lot of attention to LLMs 😉
* Lev Konstantinovskiy: Prompt Engineering 101. Introduction into LangChain — a powerful library to ease your interaction with LLMs.
* Final recommendation not from NLP: Maren Westermann: How to increase diversity in open source communities. The IT ans DS communities are diverse and spread all over the world. Let's communicate respectfully with each other!
Of course, there are way more! The whole playlist [here]😎
On the Impossible Safety of Large AI Models
The success hype of LLMs reached not only NLP-related field, but also get into life of normal humans professionals from a lot of different field. However, even I personally, have not seen any use-case where the model perform 100%, or 99.999%, or 99.9%... of the accuracy.
Theoretical proof that it is impossible to build arbitrarily accurate AI model:
https://arxiv.org/abs/2209.15259
Why? TL;DR:
* User-generated data: user-generated data are both mostly unverified and potentially highly sensitive;
* High-dimension memorization: what to achieve better score on more data? You need way more parameters. However, the contexts are limitless. So... we need infinite amount of parameters? The complexity of “fully satisfactory” language processing might be orders of magnitude larger than today’s LLMs, in which case we may still obtain greater accuracy with larger models.
* Highly heterogeneous users: the distribution of texts generated by a given user greatly diverges from the distribution of texts generated by another user. More data, more users, again, more contexts, more data which can be difficult to fully grasp and generalize.
* Sparse heavy-tailed data per user: even we take into account only one user, even their data is not so dense to be generalized. We should expect an especially large empirical heterogeneity in language data, as the samples we obtain from a user can completely stand out from the user’s language distribution.
As a result, LAIM training is unlikely to be easier than mean estimation. The usual objective for ML is to estimate a distribution which is assumed to be normal one where we want to estimate the mean. How many combinations of such distributions are we able to predict?
+ We need to find a balance between accuracy and privacy.
🤔Pretty challenging task. Will we be able to solve it anyway?
MTEB: Massive Text Embedding Benchmark
Indeed massive work of comparison of 33 models on 56 datasets and 112 languages💪
Now, if you are interested in some task, you can go to this leaderbord and orient to the best models for this task in specific language. Or, if you have new model, you can perform more clear and fair comparison.
Paper: https://arxiv.org/abs/2210.07316 (useful to read more details about the tasks, abbreviations, details of the datasets and the models)
Github: https://github.com/embeddings-benchmark/mteb
Leaderboard at 🤗: https://huggingface.co/spaces/mteb/leaderboard
DALLE-2 Without Waitlist
https://openai.com/blog/dall-e-now-available-without-waitlist/
Meet Beemo — Benchmark of expert-edited machine-generated outputs
Continuing the topic of AI-generated texts detection:
Colleagues from University of Oslo, MIT Lincoln Laboratory, Penn State University, and Toloka designed a novel benchmark of 2195 texts generated by ten instruction-finetuned language models (LMs) and edited by expert annotators for various use cases, ranging from creative writing to text summarization. Can it break all current AI-generated texts detectors? Yes!
Main info of the dataset:
😛Language: English
🤖Models: Mixtral, Mistral, LLaMa, Gemma, TULU, Zephyr.
✍️Edits then done by annotators from Toloka.ai platform.
🤗https://huggingface.co/datasets/toloka/beemo
Can you build a classifier that will detect these AI-generated texts?
Contacts:
Vladislav Mikhailov (vladism@ifi.uio.no)
Ekaterina Artemova (katya-art@toloka.ai)
Second Call for Papers for NLP4PI Workshop
Direct ARR Submission: link
Deadline: August 15th
Previous ARR Cycles Commitment: link
Deadline: August 20th
Notification of Acceptance: September 10, 2024
Join our workshop to explore how cutting-edge NLP technologies can drive social impact and support UN sustainability goals, addressing critical issues like poverty, healthcare, and climate change. We welcome submissions on innovative applications and interdisciplinary collaborations, with a special focus on solutions to combat digital violence in online spaces. Connect with NGO representatives and share your impactful research!
TextDetox CLEF 2024: Test Phase
Our shared task on multilingual text detoxification is ongoing and reaching its final phase😉
We are releasing the parallel pairs for the dev part:
https://huggingface.co/datasets/textdetox/multilingual_paradetox
and new toxic sentences for the test part:
https://huggingface.co/datasets/textdetox/multilingual_paradetox_test
We are waiting for you submission here:
https://codalab.lisn.upsaclay.fr/competitions/18243
till May 12th🤗
You can submit for ANY language! There are 9 of them: English, Spanish, German, Chinese, Arabic, Hindi, Ukrainian, Russian, and Amharic.
A little guide to building Large Language Models in 2024
by Thomas Wolf 🤗
Video [link]
Presentation [link]
TextDetox CLEF 2024
We are glad to invite you to participate in the first of its kind multilingual Text Detoxification shared task!
https://pan.webis.de/clef24/pan24-web/text-detoxification.html
TL;DR
Task formulation: transfer a text style from toxic to neutral (i.e. what a f**k is this about? -> what is this about?)
9 Languages: English, Spanish, Chinese, Hindi, Arabic, German, Russian, Ukrainian, and Amharic
🤗 https://huggingface.co/textdetox
More details:
Identification of toxicity in user texts is an active area of research. Today, social networks such as Facebook, Instagram are trying to address the problem of toxicity. However, they usually simply block such kinds of texts. We suggest a proactive reaction to toxicity from the user. Namely, we aim at presenting a neutral version of a user message which preserves meaningful content. We denote this task as text detoxification.
In this competition, we suggest you create detoxification systems for 9 languages from several linguistic families. However, the availability of training corpora will differ between the languages. For English and Russian, the parallel corpora of several thousand toxic-detoxified pairs (as presented above) are available. So, you can fine-tune text generation models on them. For other languages, for the dev phase, no such corpora will be provided. The main challenge of this competition will be to perform both supervised and unsupervised cross-lingual detoxification.
You are very welcome to test all modern LLMs on text detoxification and safety with our data as well as experiment with different unsupervised approaches based on MLMs or other paraphrasing methods!
The final leaderboard will be built on a manual evaluation of a test set subset performed via crowdsourcing at Toloka.ai platform.
In the end, you will have an opportunity to write and then present a paper at CLEF 2024 (https://clef2024.imag.fr/) which will take place in Grenoble, France!
Important Dates
February 1, 2024: First data available and run submission opens.
April 22, 2024: Registration closes.
May 6, 2024: Run submission deadline and results out.
May 31, 2024: Participants paper submission.
July 8, 2024: Camera-ready participant papers submission.
September 9-12, 2024: CLEF Conference in Grenoble and Touché Workshop.
Artificial Intelligence 2023 Playlist
Stanford series that brought together Chris Manning, Andrew Ng, Fei-Fei Li, and other researchers from Stanford to discuss the state of NLP:
https://youtube.com/playlist?list=PLoROMvodv4rPEjA3yzoqkq3J321MfH7FZ&si=eUEZC-4K3X0Ap074
I recommend for casual watching at times you are asking yourself "What's next?"
Especially:
* Chris Manning and Endrew Ng discussion about NLP.
* Andrew Ng and Fei-Fei li discussion about human-centered AI.
Ukrainian Toxicity Classification
I am glad to announce the first of its kind dataset for detection toxicity in Ukrainian🇺🇦 (~20k rows):
https://huggingface.co/datasets/ukr-detect/ukr-toxicity-dataset
Together with fine-tuned on it xlm-roberta-base:
https://huggingface.co/ukr-detect/ukr-toxicity-classifier
Happy to contribute to Ukrainian NLP💪
The work is done together with the amazing Masters student Valeriia Khylenko!
A Benchmark Dataset to Distinguish Human-Written and Machine-Generated Scientific Papers
SCIENTISTS ARE GOING TO SUBMIT PAPERS WRITTEN BY CHATGPT, THE SCIENCE GONNA DIE
Or not?
Our chair work about if we can detect machine-generated or paraphrased articles.
TL;DR: yes, we can, even with logistic regression.
For generation, we tried out: GPT-2, GPT-3, ChatGPT, Galactica, and SciGen.
Article looks like: Abstract + Intro + Conclusion.
🤗dataset with ~70k rows of generated scientific texts by different models;
There, you can also find fine-tuned 🤗Galactica and 🤗RoBERTa for detection.
The full paper with all tables of results and explainability investigations [link]
A PhD Student’s Perspective on Research in NLP in the Era of Very Large Language Models
As our IFAN project was recommended as one of the promising research direction, I will also recommend in return to read the recent paper to answer the question: "So what now in NLP research if ChatGPT is out?"
Spoiler: the world has not ended and we still have plenty work to do!
https://arxiv.org/abs/2305.12544
From my research work and what I also want to explore, my top list of research directions:
1. Misinformation fight. There is still zero online working automated fake news and propaganda detection systems. However, the risk of misinformation spread is increasing.
2. Multilingualism. A usual reminder, that there is more languages rather then English. Like at least 7k more.
3. Explainability and Interpretabilty. Do we trust models' decisions? Still absolutely far away from 100%. We can help to integrate these models into decisions making process only if their behavior will be transparent. And now think about if we can even explain every NLP task. The methods are absolutely different.
4. Less resources. Less memory to store models and fine-tune them. Less also data to learn! Do we need indeed all these training samples? Or we just need diverse enough data?
5. Human-NLP models interaction. What we can admit is that ChatGPT was the first NLP model used not only by specialists but by everyone. Because it is more or less pleasant and safe to use it. If the model cannot answer some input, it provides anyway nicely written answer. The wrapper is also extremely important. How we need to cover those models that user will be comfortable to work with it? What about children if we want to adjust them for education even from early ages?
Be brave, be creative, be inspired✨
Scaling Instruction-Finetuned Language Models
TL;DR Additional fine-tuning of T5 or PaLM models on 1k (!) tasks make them better on evaluation tasks, make them to cover more languages, and scale to the new unseen tasks better.
Google Brain team experimented with new methods of fine-tuning of Large Language Models. The main recipes for better LLMs:
* the bigger amount of the tasks for pre-training you have, the better;
* smarter prompts are also help more. By smarter here we can understand the usage of instructions and Chain-of-thought (see screenshots). Translating to human language, the more clues you give the model in the request, the more precise answer you will receive. The Chain-of-thought concept is quite interesting, the original paper of it is here.
The optimal amount of tasks of pre-training is still an open research question (authors in their experiments jumped from 282 tasks directly to 1,836 tasks, quite a gap of number to explore).
But, in the end, if we want to solve a new task and we generate smarter prompts for it, as the model was pre-trained, it will significantly improve zero-shot performance.
The original paper with all details and a lot of table and examples of performances on different tasks.
🤗model cards: all variations of t5, flan-t5-base for illustration.
Method for Fighting Harmful Multilingual Content
PhD work by Daryna Dementieva:
1. Fake News Detection using Multilingual Evidence
2. Texts Detoxification
https://www.skoltech.ru/app/data/uploads/2022/09/thesis5.pdf
Dataset No Language Left Behind
Allen AI reproduced dataset which was used for training NLLB translation model by Meta AI.
450Gb of parallel data for 200 languages are available online in 🤗:
https://huggingface.co/datasets/allenai/nllb
The paper with detaiped description:
https://arxiv.org/pdf/2207.04672.pdf