Recommendation model training

To make training more efficient, Meta’s request-only research examines recommendation model training that groups together all the items shown in response to one person’s request, rather than treating each item as a separate example.2 Meta’s research reports that this grouping reduces repeated storage and computation, making it practical to learn from more data and use more complex models to capture people’s interests.2 Meta’s research treats the organization of examples, the system that processes them and the model’s design as parts of one approach, reporting improvements in recommendation quality.2

For the delivery of training examples, Meta’s data-ingestion research examines how storing, preparing and delivering examples can limit the amount and speed of recommendation model training.3 Meta’s research describes separating data storage from a service that reads and transforms the examples, and planning computing, network and storage capacity around measurements of datasets and their preparation requirements.3 Meta’s research also describes copying datasets to the locations where training runs and identifies an opportunity for a system that assigns training jobs across regions to use capacity more efficiently.3

For the computation itself, Meta’s engineering team describes reducing the computing cost of training GEM, its ads recommendation model, by using calculations that represent numbers with less precision.1 Meta’s engineering team reports that unusually large values and errors introduced by rounding can make these calculations unreliable, so its methods spread out extreme values and use randomized rounding to avoid a consistent rounding bias.1 Meta’s engineering team also describes keeping higher precision for selected training calculations and sensitive later parts of the model when lower precision would not meet its quality targets.1

To reuse learning across recommendation surfaces, Meta’s foundation-expert research examines how improvements in one large recommendation model transfer to smaller models serving different places where recommendations appear.4 Meta’s research describes the large model producing a numerical representation of a person’s interest in a candidate item from their interaction history, which smaller models use alongside their own information.4 Meta’s research also tests whether its representations help with tasks the large model was not explicitly trained to perform.4

When trained models make live predictions, Meta’s memory-layer research examines a mismatch in recommendation systems between the numerical descriptions of items used during training and those saved for making fast, live predictions.5 Meta’s research describes training a model together with its internal store of item descriptions, so training writes the information that live predictions later read.5 Meta’s research reports deployment on Instagram Reels, with fresher item descriptions and prediction coverage rising from 96% to 100% because the design also supplies predictions for items not yet stored, including newly created Reels.5 Meta’s research also reports replacing three separate paths for passing updates from training to live prediction with one self-contained process.5