An in person audit is expensive, so it should be saved for situations where documents alone are not enough to judge the risk. Common triggers are high risk work, a serious incident or pattern of incidents, a poor document assessment that the vendor cannot explain or a new vendor taking on safety critical work for the first time.
It also makes sense to audit when the documents look fine but performance on site suggests otherwise, since paperwork and practice do not always match. Deciding these triggers ahead of time keeps audits focused on the vendors where they actually change the risk picture.