19th century engraving of the Rosetta Stone (Public Domain)

Natural language processing (NLP) has become a super-hot topic. One of the most successful applications of NLP is Machine Translation (MT) — the ability to automatically translate text from one language to another. Perhaps the most well-known MT application is the very useful Google Translate, with its ability to translate from and to 100+ languages.

How can we evaluate the performance of an MT system? One obvious way is to use human evaluation: we can show the original sentence with its machine translation side-by-side, and ask humans to rate the accuracy of the translation. Human evaluation usually requires experts who…


In the escalating battle between online video-communication services, Google Meet recently introduced a beautiful Background Blur function to compete with Zoom’s backgrounds. Google Meet’s Blur function works great when you have the right hardware and software. So, I recently tried to use Background Blur on a Surface Pro X, Microsoft’s rather fantastic 2-in-1. Alas, the Blur icon didn’t show up. The option didn’t show up:

No Background Blur :(

Why? Well, Background Blur needs an up-to-date, 64-bit Google Chrome browser. But currently there is no 64-bit Google Chrome version that can run on the Surface Pro X. No worries: enter Microsoft Edge. It’s a…


Image by Michal Jarmoluk from Pixabay

High Quality NLP Data Labeling Using Social Media Interactions

This story is based on the paper “Reactive Supervision: A New Method for Collecting Sarcasm Data” by Shmueli et al., EMNLP 2020.

Machine learning models are as good as the quality of their training data. Noisy, inaccurate labels can lead to disastrous predictions. But getting sufficient data with high quality labels is often the most challenging part of the machine learning pipeline. For NLP tasks in particular, getting labeled or annotated texts can be a messy and expensive ordeal.

For example, say you need labeled data to train a classifier for sarcasm detection. Sarcasm is a form of insincere speech…


A screenshot from the Hedy Lamarr Google Doodle

Those nifty little Google Doodles that you often see on Google’s home page may seem like nothing but harmless fun, right? I decided to dig a little deeper, and found a captivating tale that combines art and data with gender and change. But perhaps most importantly, I now realize why we need to be very, very careful about what we measure, as trusting the wrong numbers can have a devastating impact.

Behind the scenes, Google Doodles is a carefully-managed, highly-calibrated operation.

In 2014, a group of girl activists found out that Google produced many more Doodles depicting men than Doodles…


Photo by Kolar.io on Unsplash

This is a very short post about a rather hidden Google Drive feature that is very handy: adding a file to more than one folder. I use it regularly to organize my files on Google Drive, and found it to be very useful. I admit: calling this feature a “secret” is a bit on the hyperbolic side, but the truth is that this feature is not displayed in any of the online menus. Thus, you wouldn’t know of its existence unless someone told you about it! And this is why I wrote the post :)

OK, so take a quick…


“I NEED MORE RAM!” (Photo Credit: Blake Patterson)

Google Colab is an incredible tool. It can be described in three words: Python in the Cloud. OK, four words. But the important thing is, you don’t need to worry about the Python version (Colab has both Python 2 and Python 3), or about downloading modules (the most popular ones are already installed), backups, file systems, and all that administrative stuff that can sometimes cause you to kill hours on Stack Overflow. Better keep that extra energy for coding.

Google Colab is great because it simply works. It’s fantastic for learning Python, for small toy projects, but also some serious…


Metrics

Measure The Agreement Between Predicted and True Values

In my previous Multi-Class Metrics Made Simple posts, I wrote about Precision and Recall, as well as the F1-score. I received encouraging feedback from many readers. So first, thank you! In this post, I write about another popular measure: the kappa score. You might find the kappa score to be useful in your application.

The kappa score is an interesting metric. Its origins are in the field of psychology: it is used for measuring the agreement between two human evaluators or raters (e.g., psychologists) when rating subjects (patients). It was later “appropriated” by the machine-learning community to measure classification performance…


Congratulations! You’ve built a binary classifier —a fancy-schmancy neural network using 128 GPUs with their dedicated power station, or perhaps a robust logistic regression model that runs on your good old ThinkPad. You’ve designed the model and fed the data; now the time has finally come to measure the classifier’s performance.

Don’t get me wrong: ROC curves are the best choice for comparing models. However, scalar metrics still remain popular among the machine-learning community with the four most common being accuracy, recall, precision, and F1-score. Scalar metrics are ubiquitous in textbooks, web articles, online courses, and they are the metrics…


In Part I of Multi-Class Metrics Made Simple, I explained precision and recall, and how to calculate them for a multi-class classifier. In this post I’ll explain another popular performance measure, the F1-score, or rather F1-scores, as there are at least 3 variants. I’ll explain why F1-scores are used, and how to calculate them in a multi-class setting.

But first, a BIG FAT WARNING: F1-scores are widely used as a metric, but are often the wrong way to compare classifiers. You will often spot them in academic papers where researchers use a higher F1-score as “proof” that their model is…

Boaz Shmueli

I enjoy explaining stuff. PhD candidate at NLPSA, Academia Sinica. www.twitter.com/shmueli

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store