To see if the assessment approach is ineffective, MIT researchers designed a mechanism for modifying the original data so they may be sure which properties are crucial to the model. The next test is whether feature-attribution algorithms can appropriately identify those essential characteristics using this changed dataset. They discovered that even the most popular approaches frequently overlook significant elements in images and that some methods barely outperform a random baseline. This might have far-reaching consequences, particularly if neural networks are used in high-stakes scenarios such as medical diagnosis. The researchers plan to apply this review process in the future to look into other subtle or realistic aspects that might contribute to false associations.
Source: https://scitechdaily.com/can-we-tell-if-machine-learning-ai-models-are-working-correctly/
