Microsoft Releases UserLM-8b: A User-Simulating AI for Realistic Conversation Evaluation

Microsoft has released UserLM-8b, a specialized language model designed to simulate the user role in conversations rather than acting as an assistant. Unlike standard large language models trained to respond helpfully, UserLM-8b is trained to predict user turns in real-world conversations using a large dataset called WildChat-1M. It is intended to generate more realistic, diverse, and challenging interactions for evaluating assistant models. The model takes a task intent as input—such as solving a math problem or writing a piece of code—and then generates a first user utterance, follows up with additional user messages based on the conversation history, and signals when the conversation should end using the <|endconversation|> token. This makes it particularly useful for testing how well assistant models handle complex, evolving, and realistic user behaviors. UserLM-8b was developed by researchers at Microsoft Research, including Tarek Naous (intern, MSR Summer 2025), Philippe Laban, Wei Xu, and Jennifer Neville. The model is based on Llama3-8b-Base and was fine-tuned using full-parameter training on four NVIDIA RTX A6000 GPUs over 227 hours. The training data was a filtered version of WildChat-1M, with processing details provided in the accompanying paper. Evaluation results show that UserLM-8b outperforms both prior user simulation models and methods that use assistant models as user simulators. It achieves better distributional alignment with real user behavior, demonstrates stronger adherence to task intents, and generates more diverse conversations in terms of pacing, vocabulary, and information sharing. These qualities make it a more effective tool for stress-testing assistant models in realistic scenarios. However, the model has limitations. It can occasionally deviate from its assigned role or hallucinate extra requirements not present in the original task intent. While this can introduce useful variability in testing, it may also lead to inconsistent or misleading simulations. Users are advised to use highly specific and detailed task intents to reduce hallucinations. The model is currently optimized for English and may not perform reliably in other languages without further validation. UserLM-8b is not intended for direct user assistance. It should not be used in production or commercial applications without extensive testing. It is a research tool designed for evaluating assistant models, with potential future uses in user modeling, judge model training, and synthetic data generation. The model is available on Hugging Face with full documentation, including code examples and recommended generation configurations. Researchers are encouraged to apply guardrails such as filtering first tokens, avoiding premature conversation termination, and preventing repetition to improve performance. For feedback or concerns, users can contact the Microsoft Research team at plaban@microsoft.com. The model is released under a research-only license, with no guarantees of safety, accuracy, or suitability for real-world deployment.