We all love memes, and most of us have probably helped spread them–passing on that cute photo with the ironic text to our many friends on Facebook, Twitter, and elsewhere. But sometimes memes can be harmful, spreading falsehoods about people or organizations. And the tools that services like Facebook use to police text haven’t been able to do much about automatically rooting these out.
Now, though, Facebook says it has a new arrow in its quiver–a tool known as Rosetta that uses machine learning to better understand text embedded in imagery or videos. At its core, Rosetta was developed as a way of enabling photo search, or for making Facebook more accessible for users who are visually impaired. But as Facebook writes in a blog post about Rosetta, the new system is also able to do a better job than was previously possible in understanding harmful or false text used in memes that spread across Facebook and Instagram.
[Rosetta] extracts text from more than a billion public Facebook and Instagram images and video frames (in a wide variety of languages), daily and in real time, and inputs it into a text recognition model that has been trained on classifiers to understand the context of the text and the image together.
For now, the system has mostly been applied to still imagery, but Facebook plans to increasingly employ Rosetta to extract the meaning of text from video across all its applications. However, the technology isn’t ready to tackle all videos yet, the blog post explains. New tools are currently being worked on that the company hopes will do an even better job at that task.