Computer vision

Meta’s photo-description research describes models that identified concepts such as national monuments, food types and selfies7. To obtain information about positions and counts, Meta’s research also used an object detector trained to predict where objects appeared and assign descriptive labels to them7. Meta’s research reported that its improved automatic photo-description system reliably recognized more than 1,200 concepts, over ten times the number recognized by its original version7.

Instagram uses object recognition to produce automatic alternative text, a computer-written description of a photo2. Screen readers, tools that read on-screen text and descriptions aloud, make these descriptions available when people browse Feed, Explore and Profile2. The automatic description provides a list of items that a photo may contain, while a creator’s custom text can provide a richer description2.

Instagram’s Starbucks camera effect illustrates object tracking: according to Meta’s Newsroom, the camera followed a cup’s position and kept an added visual effect attached to it within the camera’s view1. A Meta patent about assistant systems describes broader computer-vision possibilities, including tracking people, interpreting multiple video frames, recognizing activities and responding to gestures8. Another Meta assistant patent describes using the surrounding scene, such as a home or workplace, to narrow down which objects or people the system should look for3. That Meta patent also describes supplementing visual recognition with information such as text read from an object’s label or GPS location data3.

A Meta patent describes using computer vision to identify visual concepts in images people have interacted with, then learning which concepts might appeal to users4. Meta’s patent proposes using those predictions to suggest concepts for new images, generate candidate images or estimate how likely people are to interact with an image4. Meta’s patent describes these possibilities for an online system4.