The model’s computational graph may include hidden attacker-controlled behavior.
Graph payloads can modify model behavior in subtle ways that are difficult to detect and control, allowing a model to appear to operate normally in production while implanted logic influences its output under specific conditions.
Complex graphs may appear suspicious without malicious intent. For example, in some cases a model export can introduce unusual, but benign, computational graph flow that has been seen in malicious models as a way to circumnavigate detections.
- Do not load or deploy the affected model until its integrity has been verified.
- Engage the team responsible for model development and deployment to determine whether the model has undergone unrecognized changes. If it has, and this is an internally developed model, engage the security team in order to understand the impact and determine the best way to handle the incident based on existing procedures and policy.
- Validate graph integrity by visual comparison with a known legitimate version of the model using a graph visualization tool such as Netron.
- If the detection is determined to be a true positive, the model should be discarded.
- Review the model to understand how its behavior has been altered, and validate its outputs using representative inputs to identify unexpected results and assess impact, especially if the model has already been deployed.