We all love memes, and most of us have probably helped spread them–passing on that cute photo with the ironic text to our many friends on Facebook, Twitter, and elsewhere. But sometimes memes can be harmful, spreading falsehoods about people or organizations. And the tools that services like Facebook use to police text haven’t been able to do much about automatically rooting these out.
[Rosetta] extracts text from more than a billion public Facebook and Instagram images and video frames (in a wide variety of languages), daily and in real time, and inputs it into a text recognition model that has been trained on classifiers to understand the context of the text and the image together.
For now, the system has mostly been applied to still imagery, but Facebook plans to increasingly employ Rosetta to extract the meaning of text from video across all its applications. However, the technology isn’t ready to tackle all videos yet, the blog post explains. New tools are currently being worked on that the company hopes will do an even better job at that task.