Fei Tian

I work on large-scale machine learning algorithms and infrastructure that power the majority of Meta’s revenue growth. The systems operate at trillion-parameter scale, with training spanning thousands of GPUs. My work includes generative models, sparse-model pretraining, reinforcement-learning-based post-training, and continual learning.

I am especially drawn to problems that are interesting, sometimes even mysterious, important, and hard. Examples include:

1. How do we solve the severe overfitting problem that appears when a large-scale sparse model enters its second epoch of training?
2. Why does a large production model regularly become over-calibrated for user traffic from a particular country at around 10:00 a.m. every day?
3. What is the right reinforcement-learning recipe, both algorithmically and in terms of infrastructure, when ads recommendation presents decoding patterns and challenges distinct from those of LLMs?
4. At tens of billions of examples per day, how should continual learning balance “fitting to the future” with “remembering the past” under minute-to-minute distribution shifts?

Collectively, these solutions and ML innovations have consistently delivered billions of dollars in revenue growth for Meta, making this one of the highest-ROI areas in monetization ML.

I worked on neural machine translation and sequence modeling. Looking back, several projects from this period explored questions that are now central to modern generative AI:

1. Shortly after the Transformer architecture was introduced, our team was among the earliest to scale it and bring it into production. We deployed Transformer systems in Microsoft Translator and contributed to a Chinese–English system reported to reach human-parity quality in 2018.
2. Deliberation Networks [NeurIPS 2017] allowed a model to revisit and improve its first-pass output, an early exploration of what we would now call test-time scaling.
3. We also studied reinforcement learning for language generation [EMNLP 2018], non-autoregressive decoding [AAAI 2019], and gradient-based neural architecture optimization [NeurIPS 2018].

I did not quite anticipate just how far these ideas would go. We should have extrapolated the scaling curve a little more aggressively. :)