This blog is related to our WDDD'14 paper titled: gem5, GPGPUSim, McPAT, GPUWattch, "Your favorite simulator here" Considered Harmful. Abstract is below, PDF is here, and talk slides are here. The issue reports for the 4 tools we looked are available as a tgz here.
We hope to use this forum for a discussion.
Abstract: Much as Dijkstra, in 1968, observed the dangers of relying on the go to statement, we observe that the dominant reliance on quantitative simulators is having a detrimental effect on our field. Over time, simulator tools have become more sophisticated. From the simple days of the now debunked SimpleScalar with its RUU-based OOO model with fixed DRAM latency, to the gem5+DramSim+GPGPUSim+McPAT mashup simulator, we have come a long way in what architects are claiming as validated tools. We argue, though, that new generations of simulators are often overfitted to certain benchmarks or configurations for validation and can have significant modeling errors that researchers are not aware of. Though the existence of these errors are unsurprising, they can cause unaware users to derive incorrect conclusions. Simultaneously, and even more problematic, is that reviewers demand researchers inappropriately use these tools. We enumerate eight common, but not acknowledged or recognized pitfalls of simulators or simulator use, considering four modern simulation infrastructures. We propose that the evaluation standards for a work should match it’s “footprint,” the breadth of layers which the technique affects, and conclude with our opinion on how to escape out of our field’s simulate-or-reject mindset.
We hope to use this forum for a discussion.
Abstract: Much as Dijkstra, in 1968, observed the dangers of relying on the go to statement, we observe that the dominant reliance on quantitative simulators is having a detrimental effect on our field. Over time, simulator tools have become more sophisticated. From the simple days of the now debunked SimpleScalar with its RUU-based OOO model with fixed DRAM latency, to the gem5+DramSim+GPGPUSim+McPAT mashup simulator, we have come a long way in what architects are claiming as validated tools. We argue, though, that new generations of simulators are often overfitted to certain benchmarks or configurations for validation and can have significant modeling errors that researchers are not aware of. Though the existence of these errors are unsurprising, they can cause unaware users to derive incorrect conclusions. Simultaneously, and even more problematic, is that reviewers demand researchers inappropriately use these tools. We enumerate eight common, but not acknowledged or recognized pitfalls of simulators or simulator use, considering four modern simulation infrastructures. We propose that the evaluation standards for a work should match it’s “footprint,” the breadth of layers which the technique affects, and conclude with our opinion on how to escape out of our field’s simulate-or-reject mindset.
