Skip to content

How a decision is made

Triggers set the verdict, severity ranks the content, your content rules decide first and triage decides the rest. reasons records the path taken.

A moderation answers two questions. flagged says whether a policy you asked to flag has fired: the SAFE or FLAGGED verdict. recommendation says what to do with the content: allow, review or reject. Both are computed in a fixed order from the scores, the template and the author, and reasons records every step that mattered.

They are separate answers. Content can be FLAGGED and allowed, because a rule lets trusted authors through. Content can be SAFE and sent to review, because its score sits between two thresholds.

The decision fields of a moderationJSON
{
  "flagged": true,
  "recommendation": "review",
  "severity": "critical",
  "policies": {
    "personal_data": 0.75,
    "links_contact": 0.9,
    "instructions": 0.8,
    "guideline/no-off-site-deals": 1 // sixteen more policies, all at 0, are trimmed
  },
  "triggered": ["personal_data", "guideline/no-off-site-deals", "instructions"],
  "rule": { "id": "r6", "name": "Links from new accounts", "action": "review" },
  "reasons": [
    "trigger:personal_data:flag",
    "trigger:guideline/no-off-site-deals:flag",
    "trigger:instructions:flag",
    "rule:r6"
  ]
}

This is the listing description from Moderate text: a link to another shop and a phone number, sent by an author Infrared had not seen before. Its severity is the highest there is, and still the recommendation is review, not reject. The steps below show why.

One decision, step by step
Sample data

The scores the decision reads: personal_data0.75 matchlinks_contact0.90 matchinstructions0.80guideline/no-off-site-deals1.00

The author: user_42status enabledtrust level new

  1. Triggers

    Each policy that can trigger, against its threshold.

    Flagged
    • Fired:personal_data a match
    • Fired:guideline/no-off-site-deals 1.00 at 0.70
    • Fired:instructions 0.80 at 0.70
    • Did not fire:links_contact 0.90, its trigger is on title
  2. Severity

    The highest score times its weight.

    1.00 × 1, socritical
  3. Content rules

    Top to bottom. The first match decides.

    r1 to r5 do not matchr6decidesReview
  4. Severity triage

    Only when no rule matched.

    Not asked:Skipped, would be reject
  5. client_action

    An escalation or an override sent with the request.

    Nothing to apply:None sent
  6. Partial floor

    Only when a policy did not answer.

    Nothing to lift:Every policy answered

flagged: truerecommendation: "review"reasonsin the order they were written

  • trigger:personal_data:flag
  • trigger:guideline/no-off-site-deals:flag
  • trigger:instructions:flag
  • rule:r6

Sample data

The sample moderation through each step of the decision, and what each step concluded.

The order

  1. Validation. A submission that fails it is rejected here and nothing below runs.
  2. Triggers decide flagged.
  3. Severity is computed from the scores.
  4. Content rules are checked top to bottom. The first one that matches sets the recommendation.
  5. Severity triage sets the recommendation when no rule matched.
  6. client_action from the request can change the recommendation.
  7. The partial floor applies when a policy did not answer.

Triggers decide flagged

A trigger watches one policy and has an effect, flag or review. It fires when the policy's score reaches its threshold, which is 0.70 for most policies, and on any match for the pattern policies such as personal_data and links_contact.

  • When a trigger with the flag effect fires, flagged is true. A trigger with the review effect leaves flagged alone and adds a review signal, used by triage below.
  • Every policy in trigger mode has a trigger. Where the template sets none, the policy gets one for the whole template. A policy in observe mode is scored and never fires.
  • A trigger set on a field replaces the one the policy would get by default. Unless the template also sets a trigger for the whole template, the policy then fires on that field and nowhere else.
  • A single object sent without field is held to the triggers of every field.

The sample shows the third point. links_contact scored 0.9, but the template triggers it on title only and the link was in description, so it is missing from triggered. The three policies that are listed had no trigger of their own and fired through the default one. Triggers has the rule in full.

Severity ranks the content

Severity is the highest score after each is multiplied by its policy's weight. Safety policies such as toxicity count in full, links_contact at 0.95, spam at 0.8, and quality signals such as image_quality at 0.5, unless the template sets other weights. A match on a pattern policy counts as at least 0.85 before its weight, so one link or one phone number ranks high. Observed policies do not count.

In the sample the guideline scored 1 at full weight, which makes the severity critical. The response carries the band, not the number:

severityThe weighted score is
noneexactly 0
lowbelow the review threshold
mediumat or above the review threshold, 0.40 by default
highat or above the reject threshold, 0.80 by default
critical0.95 or more

Rules first, triage for the rest

Your content rules read the scores, the author's status and trust level, the values of fields and the language. They are checked top to bottom, and the first rule whose conditions all hold sets the recommendation. rule in the response names it.

Inside a rule, "is flagged" means the score reached the policy's own threshold. It does not look at triggers, so a policy you only observe can still drive a rule. That is how the sample's rule, "Links from new accounts", saw links_contact although no trigger fired for it.

When no rule matches, severity triage decides: allow below the review threshold, review from it, reject from the reject threshold. If triage says allow and a trigger with the review effect fired, the recommendation becomes review. The basic check has no rules, so triage decides everything there.

Triage alone would have rejected the sample. The rule matched first, so it goes to a reviewer instead.

A request without author_id is judged as an author with status enabled and the lowest trust level, new.

Change the result from your side

Your backend sometimes knows something Infrared does not: the account was reported an hour ago, or a staff member is posting. Send client_action with the request to act on that.

client_action.actionstringRequired

escalate applies your recommendation only when it is stricter than the computed one. override replaces the computed one.

One ofescalateoverride

client_action.recommendationstringRequired

The recommendation you want.

One ofallowreviewreject

curl https://api.useinfrared.com/v1/moderations \
  -H "Authorization: Bearer $INFRARED_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "type": "text",
    "input": "Do you deliver to the city centre?",
    "author_id": "user_42",
    "client_action": {
      "action": "escalate",
      "recommendation": "review"
    }
  }'
200 OK, trimmedJSON
{
  "object": "moderation",
  "id": "mod_4vpwwLeQiXC0EA2C2EUdF7",
  "status": "completed",
  "flagged": false,
  "recommendation": "review",
  "severity": "none",
  "triggered": [],
  "rule": null,
  "reasons": ["triage:allow", "client_action:escalate:review"]
}

Every policy scored this message 0 and triage allowed it. The escalation then raised the recommendation to review. Only the recommendation changes: flagged, triggered and rule stay as computed, and a review result enters the review queue as usual.

When a policy did not answer

A moderation whose status is partial was decided without one or more policies. If one of them was in trigger mode and the recommendation would be allow, it becomes review, so nothing passes unscreened. A recommendation set by a content rule is left as it is.

This step runs last. When no rule decided, it also turns a client_action that overrides to allow back into review.

Every value of reasons

reasons lists the steps in the order they happened.

ValueMeaning
validation_failedValidation failed. It is the only reason on an invalid moderation.
trigger:toxicity:flagA template-wide trigger fired: the policy, then its effect, flag or review.
trigger:toxicity@description:flagA trigger set on a field fired: the policy, the field, the effect.
rule:r6The content rule with this id decided.
triage:allow, triage:review, triage:rejectNo rule matched and severity triage decided.
review_triggerA trigger with the review effect turned triage's allow into review.
client_action:escalate:reviewThe escalation in the request took effect. The last part is the new recommendation.
client_action:override:allowThe override in the request replaced the recommendation.
partial:toxicity,spamThese policies did not answer. One entry, keys separated by commas.
partial_reviewA policy that did not answer turned allow into review.

An escalation that is not stricter than the computed result changes nothing and adds no reason.

Next

Something missing or unclear on this page? Write to support@useinfrared.com.