The use of numerical methods in fire safety investigations for civil buildings and nuclear facilities has received enormous attention in recent years. To evaluate quantities—such as gas temperatures—in fire models, local metrics using single points (e.g. comparing maximum or minimum peak value of two time series) are well-established. Experimental (measurement and model input) uncertainty estimates can be used to quantify the model uncertainty. Although the peak value is a relevant and well-defined quantity, global metrics comparing the entire course of two time series can often provide additional information for the validation of fire models. A comparative methodology COMET for evaluating the predictive power of fire models is developed and presented in this paper. In the methodology, both local and global metrics are combined to incorporate the explanatory power of both quantities in the validation process. While uncertainty analysis is well established for peak values, to the best of our knowledge, there are no analytic results on quantifying the uncertainty of the global metric in the literature. We address the latter based on experimental measurements and derive confidence regions for both metrics. Finally, this paper summarizes the results using COMET to validate the Fire Dynamics Simulator (FDS) version 6 for a room fire scenario. Validation examples are tests 3, 4 and 5 of the DOOR series of the international OECD/NEA PRISME project, in which the transport of heat and flue gases through a door between two rooms was examined. Using COMET, we can easily identify sensors with high level of agreement between model and experimental results with respect to the local and/or the global metric.