Custom Detection Rules

Create and manage custom security detection rules tailored to your environment and threat landscape.

API examples: base URL is configurable

The API examples below use http://localhost:5000 (default for an on‑prem/local Worker).

If your Worker runs on a different host/port, replace the base URL accordingly.

Two Rule Types

CastellanAI supports YARA rules for file-based detection and Security Event Rules for log-based detection.

---

Overview

CastellanAI supports two types of detection rules:

Rule Type	Purpose	Management
Malware Rules (YARA)	Signature-based file scanning patterns	Dashboard UI
Security Event Rules	Map Windows Event IDs to security alerts	API (Admin only)

---

Malware Rules (YARA)

YARA rules are the primary method for creating custom file-based detections using industry-standard syntax.

Accessing Malware Rules

Navigate to Malware Analysis in the sidebar, then access the Detection Rules section.

Importing Custom YARA Rules

📥 Import Process

Import Steps

1. Click Import Rules on the Malware Analysis page
2. Upload a YARA rule file (.yar, .yara, or .txt)
3. Configure import settings:

Setting	Description
Category	Classify the rule (Malware, Ransomware, Trojan, etc.)
Author	Rule author attribution
Skip Duplicates	Avoid importing existing rules
Enable by Default	Activate rules immediately after import

4. Click Import to add the rules

📋 Rule Structure

YARA Rule Structure

rule SuspiciousPowerShell : suspicious
{
    meta:
        description = "Detects encoded PowerShell commands"
        author = "Security Team"
        severity = "high"

    strings:
        $enc1 = "-encodedcommand" nocase
        $enc2 = "-enc " nocase
        $b64 = /[A-Za-z0-9+\/]{50,}={0,2}/

    condition:
        ($enc1 or $enc2) and $b64
}

🔧 Components

Rule Components

Component	Description
Rule Name	Unique identifier (e.g., SuspiciousPowerShell)
Tags	Classification tags (e.g., : suspicious)
Meta	Metadata including description, author, severity
Strings	Patterns to match (text, hex, regex)
Condition	Logic determining when rule triggers

Managing YARA Rules

From the Malware Analysis page:

Action	Description
Enable/Disable	Toggle rule activation
View Details	See full rule content and match statistics
Delete	Remove custom rules (built-in rules cannot be deleted)
Filter	Search by name, category, threat level, or status

---

Security Event Rules

Security Event Rules map Windows Event Log entries to security alerts with MITRE ATT&CK technique mappings.

Rule Structure

📋 Fields

Rule Fields

Field	Description
Event ID	Windows Event ID (e.g., 4625 for failed logon)
Channel	Log source (e.g., "Security", "PowerShell/Operational")
Event Type	Security classification (e.g., "AuthenticationFailure")
Risk Level	Severity: low, medium, high, critical
Confidence	Detection confidence score (0-100)
Summary	Human-readable description
MITRE Techniques	Mapped ATT&CK technique IDs
Recommended Actions	Response guidance for analysts

📦 Pre-built

Pre-built Rules

CastellanAI includes 50+ pre-built Security Event Rules:

Category	Event IDs
Authentication Events	4624, 4625 (Logon success/failure)
Privilege Escalation	4672 (Special privilege assignment)
Account Management	4720, 4732 (User/group changes)
Process Execution	4104, 4698 (PowerShell, scheduled tasks)
Service Installation	7045 (New services)
Security Policy	4719 (Audit policy changes)

Creating Custom Security Event Rules

Admin Only

Custom Security Event Rules are managed via API and require Admin role.

➕ Create

Create a Rule

curl -X POST "http://localhost:5000/api/security-event-rules" \
  -H "Authorization: Bearer YOUR_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "eventId": 4688,
    "channel": "Security",
    "eventType": "ProcessCreation",
    "riskLevel": "medium",
    "confidence": 70,
    "summary": "New process created",
    "mitreTechniques": ["T1059"],
    "recommendedActions": ["Review process details", "Check parent process"],
    "isEnabled": true,
    "priority": 100,
    "description": "Monitors for suspicious process creation",
    "tags": ["execution", "custom"]
  }'

✏️ Update

Update a Rule

curl -X PUT "http://localhost:5000/api/security-event-rules/{id}" \
  -H "Authorization: Bearer YOUR_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "riskLevel": "high",
    "confidence": 85
  }'

🗑️ Delete

Delete a Rule

curl -X DELETE "http://localhost:5000/api/security-event-rules/{id}" \
  -H "Authorization: Bearer YOUR_TOKEN"

Caution

Deleting a rule removes it permanently. Consider disabling instead.

API Endpoints

Endpoint	Method	Description
/api/security-event-rules	GET	List all rules with optional filtering
/api/security-event-rules/{id}	GET	Get specific rule
/api/security-event-rules	POST	Create new rule (Admin)
/api/security-event-rules/{id}	PUT	Update rule (Admin)
/api/security-event-rules/{id}	DELETE	Delete rule (Admin)
/api/security-event-rules/refresh-cache	POST	Refresh rule cache (Admin)

---

Best Practices

📄 YARA Rules

YARA Rule Best Practices

Do	Don't
Start with narrow, specific patterns	Create overly broad rules
Test thoroughly before enabling	Enable untested rules in production
Use metadata (author, description, severity)	Leave rules without context
Leverage community rules from trusted sources	Import rules from unknown sources

📊 Event Rules

Security Event Rule Best Practices

Do	Don't
Map to MITRE ATT&CK techniques	Skip technique mappings
Set appropriate risk levels	Mark everything as critical
Include recommended actions	Leave response guidance empty
Use tags for categorization	Create unorganized rules

⚠️ Common Mistakes to Avoid

Mistake	Impact	Solution
Enabling untested rules	High false positive rate	Test in staging first
Overly broad patterns	Alert fatigue	Start specific, broaden gradually
Missing documentation	Hard to maintain	Always include description and author
Ignoring false positives	Missed real threats	Review and tune regularly
Setting everything as critical	Alert fatigue	Reserve critical for high-confidence threats

---

Availability

Feature	Small Business	Medium Business	Enterprise
Built-in YARA rules	✅	✅	✅
Import YARA rules	✅	✅	✅
Built-in Event rules	✅	✅	✅
Create Event rules via API	-	Limited	Unlimited
Rule performance analytics	-	-	✅

---

What's Next?

Guide	Description
Threat Detection	Understand detection methods
MITRE ATT&CK	Learn about technique mappings
Malware Scanning	File-based threat detection
Taking Action	Response procedures