Troubleshooting

This guide covers common issues and solutions when working with Foundatio.Repositories.

Connection Issues

Cannot Connect to Elasticsearch

Symptoms:

• No connection could be made
• Connection refused
• Timeout errors

Solutions:

1. Verify Elasticsearch is running:

curl http://localhost:9200

2. Check connection string:

protected override IConnectionPool CreateConnectionPool()
{
    // Ensure URL is correct
    return new SingleNodeConnectionPool(new Uri("http://localhost:9200"));
}

3. Check firewall/network:

# Test connectivity
telnet localhost 9200

4. Enable debug logging:

protected override void ConfigureSettings(ConnectionSettings settings)
{
    settings.DisableDirectStreaming();
    settings.PrettyJson();
    settings.EnableDebugMode();
}

Authentication Errors

Symptoms:

• 401 Unauthorized
• 403 Forbidden

Solutions:

protected override void ConfigureSettings(ConnectionSettings settings)
{
    // Basic authentication
    settings.BasicAuthentication("username", "password");
    
    // Or API key
    settings.ApiKeyAuthentication("api-key-id", "api-key");
}

Index Issues

Index Not Found

Symptoms:

• index_not_found_exception
• no such index

Solutions:

1. Configure indexes on startup:

await configuration.ConfigureIndexesAsync();

2. Check index name:

// Versioned indexes have version suffix
// "employees" -> "employees-v1"
var indexName = index.VersionedName;

3. Verify index exists:

curl http://localhost:9200/_cat/indices

Mapping Conflicts

Symptoms:

• mapper_parsing_exception
• failed to parse field

Solutions:

1. Increment index version:

// Change version to trigger reindex
public EmployeeIndex(...) : base(configuration, "employees", version: 2) { }

2. Delete and recreate index (development only):

await configuration.DeleteIndexesAsync();
await configuration.ConfigureIndexesAsync();

3. Check field types match:

// Ensure mapping matches data types
.Number(f => f.Name(e => e.Age).Type(NumberType.Integer))

Query Issues

No Results Returned

Symptoms:

• Empty results when data exists
• Total: 0

Solutions:

1. Check soft delete mode:

// Include soft-deleted documents
var results = await repository.FindAsync(query, o => o.IncludeSoftDeletes());

2. Use immediate consistency:

// Wait for index refresh
await repository.AddAsync(entity, o => o.ImmediateConsistency());
var results = await repository.FindAsync(query);

3. Verify filter syntax:

// Check filter expression
var results = await repository.FindAsync(q => q.FieldEquals(e => e.Status, "active"));

// Debug: Log the query
var results = await repository.FindAsync(query, o => o.QueryLogLevel(LogLevel.Debug));

4. Check field names:

// Use exact field names from mapping
// "name" vs "name.keyword" for exact match

Query Syntax Errors

Symptoms:

• query_parsing_exception
• failed to parse query

Solutions:

1. Escape special characters:

// Escape: + - = && || > < ! ( ) { } [ ] ^ " ~ * ? : \ /
var escaped = Regex.Escape(userInput);

2. Use strongly-typed queries:

// Instead of filter expression
var results = await repository.FindAsync(q => q
    .FieldEquals(e => e.Status, "active")
    .FieldCondition(e => e.Name, ComparisonOperator.Contains, "John"));

For numeric comparisons, use FilterExpression with Lucene syntax:

var results = await repository.FindAsync(q => q
    .FieldEquals(e => e.Status, "active")
    .FilterExpression("age:[25 TO *]"));

Cache Issues

Stale Data

Symptoms:

• Old data returned after updates
• Changes not reflected

Solutions:

1. Manually invalidate cache:

await repository.InvalidateCacheAsync(document);
await repository.InvalidateCacheAsync("custom-cache-key");

2. Disable cache for debugging:

var results = await repository.FindAsync(query, o => o.Cache(false));

3. Check cache invalidation gaps:

// PatchAllAsync doesn't invalidate custom keys
await repository.PatchAllAsync(query, patch);
await repository.InvalidateCacheAsync("affected-key");

Cache Key Conflicts

Symptoms:

• Wrong data returned
• Data from different queries mixed

Solutions:

// Use unique, consistent cache keys
var key = $"employee:email:{email.ToLowerInvariant()}";
var results = await repository.FindOneAsync(query, o => o.Cache(key));

Version Conflicts

VersionConflictDocumentException

Symptoms:

• version_conflict_engine_exception
• VersionConflictDocumentException

Solutions:

1. Implement retry logic:

int retries = 3;
while (retries > 0)
{
    try
    {
        var doc = await repository.GetByIdAsync(id);
        doc.Name = "Updated";
        await repository.SaveAsync(doc);
        break;
    }
    catch (VersionConflictDocumentException)
    {
        retries--;
        if (retries == 0) throw;
    }
}

2. Skip version check (if appropriate):

await repository.SaveAsync(document, o => o.SkipVersionCheck());

3. Use atomic operations:

// Atomic increment avoids conflicts
await repository.PatchAsync(id, new ScriptPatch("ctx._source.counter++"));

Performance Issues

Slow Queries

Symptoms:

• High query latency
• Timeouts

Solutions:

1. Add appropriate indexes:

// Ensure fields are properly mapped
.Keyword(f => f.Name(e => e.Status))  // For filtering
.Text(f => f.Name(e => e.Name).AddKeywordAndSortFields())  // For search + sort

2. Limit result size:

var results = await repository.FindAsync(query, o => o.PageLimit(100));

3. Use field selection:

var results = await repository.FindAsync(query, o => o
    .Include(e => e.Id)
    .Include(e => e.Name));

4. Use search-after for deep pagination:

var results = await repository.FindAsync(query, o => o.SearchAfterPaging());

Memory Issues

Symptoms:

• OutOfMemoryException
• High memory usage

Solutions:

1. Use batch processing:

await repository.BatchProcessAsync(query, async batch =>
{
    // Process in batches
    return true;
}, o => o.PageLimit(500));

2. Use snapshot paging for large exports:

var results = await repository.FindAsync(query, o => o.SnapshotPaging());

Notification Issues

EntityChanged Not Received

Symptoms:

• Subscribers not receiving notifications
• Message bus appears silent

Solutions:

1. Verify message bus is configured:

public MyElasticConfiguration(IMessageBus messageBus, ...) 
    : base(messageBus: messageBus, ...) { }

2. Check notifications are enabled:

// Repository level
NotificationsEnabled = true;

// Operation level
await repository.SaveAsync(entity, o => o.Notifications(true));

3. Verify subscription:

await messageBus.SubscribeAsync<EntityChanged>(async (msg, ct) =>
{
    Console.WriteLine($"Received: {msg.Type} {msg.ChangeType}");
});

Soft Delete Not Sending Removed

Symptoms:

• Soft delete sends ChangeType.Saved instead of Removed

Solutions:

// Enable originals tracking
public class EmployeeRepository : ElasticRepositoryBase<Employee>
{
    public EmployeeRepository(EmployeeIndex index) : base(index)
    {
        OriginalsEnabled = true;  // Required for soft delete detection
    }
}

Debugging Tips

Enable Detailed Logging

// In configuration
protected override void ConfigureSettings(ConnectionSettings settings)
{
    settings.DisableDirectStreaming();
    settings.PrettyJson();
    settings.OnRequestCompleted(details =>
    {
        _logger.LogDebug("Request: {Method} {Uri}", 
            details.HttpMethod, details.Uri);
        if (details.RequestBodyInBytes != null)
            _logger.LogDebug("Body: {Body}", 
                Encoding.UTF8.GetString(details.RequestBodyInBytes));
    });
}

// Per query
var results = await repository.FindAsync(query, o => o.QueryLogLevel(LogLevel.Debug));

Inspect Elasticsearch Directly

# Check cluster health
curl http://localhost:9200/_cluster/health

# List indexes
curl http://localhost:9200/_cat/indices

# View mapping
curl http://localhost:9200/employees/_mapping

# Search directly
curl -X POST http://localhost:9200/employees/_search -H 'Content-Type: application/json' -d '
{
  "query": { "match_all": {} }
}'

Check Index Statistics

curl http://localhost:9200/employees/_stats

Common Error Messages

Error	Cause	Solution
index_not_found_exception	Index doesn't exist	Run ConfigureIndexesAsync()
mapper_parsing_exception	Type mismatch	Check field types in mapping
version_conflict_engine_exception	Concurrent modification	Implement retry or skip version check
search_phase_execution_exception	Query error	Check query syntax
circuit_breaking_exception	Memory limit	Reduce batch size
cluster_block_exception	Cluster read-only	Check disk space

Repository Exception Types

Foundatio.Repositories uses typed exceptions so callers can handle specific failure modes. The exceptions listed below inherit from DocumentException.

Exception	When Thrown	Retryable?
DuplicateDocumentException	AddAsync when a document with the same ID already exists	No — remove the existing document or use SaveAsync
VersionConflictDocumentException	SaveAsync / PatchAsync when the document version doesn't match (HTTP 409)	Yes — re-fetch the document and retry
DocumentNotFoundException	PatchAsync when the target document doesn't exist (HTTP 404)	No — verify the document ID
DocumentValidationException	Any write operation when document validation fails	No — fix the document data
DocumentException	Other Elasticsearch errors not covered above	Depends on the underlying cause

Partial Failures on Bulk Operations

When AddAsync or SaveAsync is called with multiple documents, some may succeed and others may fail. The repository:

1. Processes all successes first — fires events, populates cache, sends notifications.
2. Leaves failed documents' cache unchanged — failed writes don't mutate Elasticsearch, so existing cache entries remain valid. The writer that caused a conflict handles its own cache update via message bus notifications.
3. Throws a typed exception — DuplicateDocumentException for add, VersionConflictDocumentException for save.

try
{
    await repository.AddAsync(documents);
}
catch (DuplicateDocumentException ex)
{
    // Successful documents were fully processed.
    // Duplicate documents: existing cache entries preserved (nothing was mutated).
    _logger.LogWarning(ex, "Partial failure: some documents already existed");
}
catch (VersionConflictDocumentException ex)
{
    _logger.LogWarning(ex, "Partial failure: some documents had version conflicts");
}

Transient Error Retries

The repository automatically retries transient Elasticsearch errors:

• HTTP 429 (Too Many Requests) — retried with exponential backoff, up to 3 retries (4 total attempts)
• HTTP 503 (Service Unavailable) — retried with exponential backoff, up to 3 retries (4 total attempts)
• HTTP 409 (Version Conflict) — not retried; the caller must handle conflict resolution
• DuplicateDocumentException — not retried by the resilience policy

Aggregation Warnings

doc_count_error_upper_bound Warning

Symptoms:

• Warning-level log message about doc_count_error_upper_bound in terms aggregation results

Explanation:

When running terms aggregations across multiple shards, Elasticsearch returns an approximate count. The doc_count_error_upper_bound field indicates the maximum potential error in document counts for each term bucket. A non-zero value means shard-level approximations may have affected the results.

Solutions:

1. Increase shard_size if accuracy matters for your use case — this makes Elasticsearch consider more terms per shard before combining results
2. Use a single shard for small indexes where exact counts are important
3. Ignore the warning if approximate counts are acceptable for your use case (this is common for analytics and dashboards)

Getting Help

1. Check logs - Enable debug logging
2. Inspect Elasticsearch - Use Kibana or curl
3. Review documentation - Check specific feature guides
4. GitHub Issues - Search existing issues or create new one
5. Discord - Join the Foundatio Discord community

Next Steps

• Configuration - Configuration options
• Elasticsearch Setup - Connection setup
• Caching - Cache troubleshooting