AI Moderation

Screen text, image and video messages for unsafe content automatically in your Android app, and respond when each one is approved or rejected.

​

Overview

AI Moderation in the CometChat SDK helps ensure that your chat application remains safe and compliant by automatically reviewing messages for inappropriate content. This feature leverages AI to moderate messages in real-time, reducing manual intervention and improving user experience.

For a broader understanding of moderation features, configuring rules, and managing flagged messages, see the Moderation Overview.

​

Prerequisites

Before using AI Moderation, ensure the following:

1. Moderation is enabled for your app in the CometChat Dashboard
2. Moderation rules are configured under Moderation > Rules
3. You’re using CometChat SDK version that supports moderation

​

How It Works

Step	Description
1. Send Message	App sends a text, image, or video message
2. Pending Status	Message is sent with PENDING moderation status (getModerationStatus()). This is separate from the send status: with offline support on, onMessagePending fires first, before the message reaches the server.
3. AI Processing	Moderation service analyzes the content
4. Result Event	onMessageModerated event fires with final status

​

Supported Message Types

Moderation is triggered only for the following message types:

Message Type	Moderated	Notes
Text Messages	✅	Content analyzed for inappropriate text
Image Messages	✅	Images scanned for unsafe content
Video Messages	✅	Videos analyzed for prohibited content
Custom Messages	❌	Not subject to AI moderation
Action Messages	❌	Not subject to AI moderation

​

Moderation Status

The getModerationStatus() method returns one of the following values:

Status	Enum Value	Description
Pending	ModerationStatus.PENDING	Message is being processed by moderation
Approved	ModerationStatus.APPROVED	Message passed moderation and is visible
Disapproved	ModerationStatus.DISAPPROVED	Message violated rules and was blocked

​

Implementation

​

Step 1: Send a Message and Check Initial Status

When you send a text, image, or video message, check the initial moderation status:

Starting with v5.1.0, the CallbackListener overloads of `sendMessage()` are deprecated. Use MessageSentListener instead; it requires SDK 5.1.0 or later. On 5.0.x, keep using CallbackListener and handle the sent message in onSuccess where these examples use onMessageSent. With offline support off, it behaves exactly like the old callback: onError is unchanged, and the sent message arrives in onMessageSent instead of onSuccess. With offline support on, onMessagePending fires first. A message sent without a connection stays PENDING and is sent when the connection returns: onError fires with code ERROR_INTERNET_UNAVAILABLE as a not-connected notice, and onMessageSent follows once the message is sent. In onError, check the message’s getStatus(): PENDING means queued, FAILED means it failed. The old CallbackListener overloads are never queued; offline, they end with onError. See Send While Offline.

Kotlin

val textMessage = TextMessage(receiverUID, "Hello, how are you?", CometChatConstants.RECEIVER_TYPE_USER)

CometChat.sendMessage(textMessage, object : CometChat.MessageSentListener() {
    override fun onMessagePending(message: BaseMessage?) {
        // Fires once the message is queued while offline support is enabled; message?.status == MessageStatus.PENDING
    }

    override fun onMessageSent(message: BaseMessage?) {
        // Check moderation status
        if ((message as? TextMessage)?.moderationStatus == ModerationStatus.PENDING) {
            Log.d(TAG, "Message is under moderation review")
            // Show pending indicator in UI
        }
    }

    override fun onError(e: CometChatException) {
        // With offline support on, this also fires when a message is queued offline: check the message's status (PENDING = queued, FAILED = failed).
        Log.e(TAG, "Message sending failed: ${e.message}")
    }
})

Java

TextMessage textMessage = new TextMessage(receiverUID, "Hello, how are you?", CometChatConstants.RECEIVER_TYPE_USER);

CometChat.sendMessage(textMessage, new CometChat.MessageSentListener() {
    @Override
    public void onMessagePending(BaseMessage message) {
        // Fires once the message is queued while offline support is enabled; message.getStatus() == MessageStatus.PENDING
    }

    @Override
    public void onMessageSent(BaseMessage message) {
        // Check moderation status
        if (message instanceof TextMessage
                && ((TextMessage) message).getModerationStatus().equals(ModerationStatus.PENDING)) {
            Log.d(TAG, "Message is under moderation review");
            // Show pending indicator in UI
        }
    }

    @Override
    public void onError(CometChatException e) {
        // With offline support on, this also fires when a message is queued offline: check the message's status (PENDING = queued, FAILED = failed).
        Log.e(TAG, "Message sending failed: " + e.getMessage());
    }
});

​

Step 2: Listen for Moderation Results

Register a message listener to receive moderation results in real-time:

Kotlin

val listenerID = "MODERATION_LISTENER"

CometChat.addMessageListener(listenerID, object : CometChat.MessageListener() {
    override fun onMessageModerated(message: BaseMessage) {
        when (message) {
            is TextMessage -> {
                when (message.moderationStatus) {
                    ModerationStatus.APPROVED -> {
                        Log.d(TAG, "Message ${message.id} approved")
                        // Update UI to show message normally
                    }
                    ModerationStatus.DISAPPROVED -> {
                        Log.d(TAG, "Message ${message.id} blocked")
                        // Handle blocked message (hide or show warning)
                    }
                }
            }
            is MediaMessage -> {
                when (message.moderationStatus) {
                    ModerationStatus.APPROVED -> {
                        Log.d(TAG, "Media message ${message.id} approved")
                    }
                    ModerationStatus.DISAPPROVED -> {
                        Log.d(TAG, "Media message ${message.id} blocked")
                    }
                }
            }
        }
    }
})

// Don't forget to remove the listener when done
// CometChat.removeMessageListener(listenerID)

Java

String listenerID = "MODERATION_LISTENER";

CometChat.addMessageListener(listenerID, new CometChat.MessageListener() {
    @Override
    public void onMessageModerated(BaseMessage message) {
        if (message instanceof TextMessage) {
            TextMessage textMessage = (TextMessage) message;
            if (textMessage.getModerationStatus().equals(ModerationStatus.APPROVED)) {
                Log.d(TAG, "Message " + message.getId() + " approved");
                // Update UI to show message normally
            } else if (textMessage.getModerationStatus().equals(ModerationStatus.DISAPPROVED)) {
                Log.d(TAG, "Message " + message.getId() + " blocked");
                // Handle blocked message (hide or show warning)
            }
        } else if (message instanceof MediaMessage) {
            MediaMessage mediaMessage = (MediaMessage) message;
            if (mediaMessage.getModerationStatus().equals(ModerationStatus.APPROVED)) {
                Log.d(TAG, "Media message " + message.getId() + " approved");
            } else if (mediaMessage.getModerationStatus().equals(ModerationStatus.DISAPPROVED)) {
                Log.d(TAG, "Media message " + message.getId() + " blocked");
            }
        }
    }
});

// Don't forget to remove the listener when done
// CometChat.removeMessageListener(listenerID);

​

Step 3: Handle Disapproved Messages

When a message is disapproved, handle it appropriately in your UI:

Kotlin

fun handleDisapprovedMessage(message: BaseMessage) {
    val messageId = message.id

    // Option 1: Hide the message completely
    hideMessageFromUI(messageId)

    // Option 2: Show a placeholder message
    showBlockedPlaceholder(messageId, "This message was blocked by moderation")

    // Option 3: Notify the sender (if it's their message)
    if (message.sender.uid == currentUserUID) {
        showNotification("Your message was blocked due to policy violation")
    }
}

Java

private void handleDisapprovedMessage(BaseMessage message) {
    int messageId = message.getId();

    // Option 1: Hide the message completely
    hideMessageFromUI(messageId);

    // Option 2: Show a placeholder message
    showBlockedPlaceholder(messageId, "This message was blocked by moderation");

    // Option 3: Notify the sender (if it's their message)
    if (message.getSender().getUid().equals(currentUserUID)) {
        showNotification("Your message was blocked due to policy violation");
    }
}

​

Next Steps

After implementing AI Moderation, consider adding a reporting feature to allow users to flag messages they find inappropriate. For more details, see the Flag Message documentation.