Search, Filtering & Pagination Concepts

As your parcel tracking API grows, clients need ways to find specific parcels among thousands of records. Returning every parcel on every request is not practical. This lesson covers the concepts behind dynamic filtering, pagination strategies, and response caching that you will implement in the following presentations.

Why Search and Filtering Matter

A parcel tracking system accumulates data quickly. A medium-sized carrier might process 10,000 parcels per day. Without filtering, a "list all parcels" endpoint would return enormous payloads, waste bandwidth, and strain the database.

Effective search and filtering let clients:

• Narrow results to a specific subset (e.g., all parcels with status InTransit)
• Combine criteria to answer business questions (e.g., all Express parcels shipped to Berlin in the last week)
• Search by keyword across multiple fields (e.g., find a parcel by partial tracking number or description)
• Reduce response size so mobile and low-bandwidth clients stay responsive

Dynamic Filtering with IQueryable

In EF Core, IQueryable<T> represents a query that has not been executed yet. You can compose filters onto it conditionally, and EF Core translates the entire chain into a single SQL query.

The key idea is deferred execution: calling .Where() does not hit the database. The query runs only when you materialize results with ToListAsync(), CountAsync(), or similar operators.

This makes IQueryable perfect for dynamic filtering. You start with the base query and conditionally append .Where() clauses based on which query parameters the client provided:

1Base query: dbContext.Parcels
2  + if status provided:        .Where(p => p.Status == status)
3  + if serviceType provided:   .Where(p => p.ServiceType == serviceType)
4  + if dateFrom provided:      .Where(p => p.CreatedAt >= dateFrom)
5  + if keyword provided:       .Where(p => p.TrackingNumber.Contains(keyword) || ...)
6  = Final SQL with only relevant WHERE clauses

Each .Where() call adds an AND condition. When a parameter is not provided, you simply skip that filter, and it does not appear in the generated SQL. The database only processes the conditions that matter for each request.

Pagination Strategies

Pagination controls how many results are returned per request and how the client navigates through pages. There are two main approaches.

Offset-Based Pagination

Offset pagination uses Skip(n) and Take(pageSize) to select a window of results. The client passes a page number, and the server calculates the offset.

1Page 1: OFFSET 0  FETCH 20
2Page 2: OFFSET 20 FETCH 20
3Page 3: OFFSET 40 FETCH 20

Advantages:

• Simple to implement and understand
• Clients can jump to any page directly

Disadvantages:

• Performance degrades on large offsets because the database must scan and discard rows
• Inconsistent results when data changes between page requests (inserts or deletes shift rows)

Cursor-Based Pagination

Cursor pagination uses a value from the last returned record as a reference point. Instead of "skip 200 rows," the client says "give me 20 rows after this cursor."

The cursor is typically the primary key or a combination of sort key and primary key, encoded as an opaque string. The server decodes it and uses a WHERE clause to start from the right position.

1Request 1: Give me 20 parcels (no cursor)
2Response:  [parcels 1-20], nextCursor = "abc123"
3
4Request 2: Give me 20 parcels after cursor "abc123"
5Response:  [parcels 21-40], nextCursor = "def456"

Advantages:

• Consistent performance regardless of how deep into the dataset you are
• Stable results even when data changes between requests
• Well-suited for "load more" and infinite scroll UIs

Disadvantages:

• Cannot jump to an arbitrary page
• Slightly more complex to implement

For the Parcel Tracking API, we use cursor-based pagination because parcels are frequently created and updated, making offset pagination unreliable.

Sorting

Sorting determines the order of results before pagination is applied. Without a stable sort order, pagination produces unpredictable results.

Common sort fields for parcels include:

A stable sort requires a tiebreaker. When two parcels have the same CreatedAt timestamp, the sort order is ambiguous. Adding the primary key (Id) as a secondary sort guarantees deterministic ordering:

ORDER BY CreatedAt DESC, Id DESC

This tiebreaker is essential for cursor-based pagination to work correctly. The cursor encodes both the sort field value and the Id, so every row has a unique position in the result set.

Page Size Limits

Clients should be able to request their preferred page size, but the server must enforce boundaries:

• Default page size: 20 items (reasonable for most clients)
• Maximum page size: 100 items (prevents accidental or malicious large queries)
• Minimum page size: 1 item (at least one result per page)

If the client requests a page size outside these bounds, the server clamps it to the nearest valid value without returning an error.

Pagination Metadata

Every paginated response should include metadata that tells the client about the result set:

json
1{
2  "items": [...],
3  "totalCount": 1547,
4  "pageSize": 20,
5  "cursor": "eyJpZCI6IjEyMyJ9",
6  "nextCursor": "eyJpZCI6IjE0MyJ9",
7  "hasNextPage": true
8}

• totalCount: The total number of items matching the filters (before pagination)
• pageSize: The actual page size used for this request
• cursor: The cursor for the current page (useful for refreshing)
• nextCursor: The cursor to fetch the next page (null if no more pages)
• hasNextPage: A boolean flag so clients do not need to check if nextCursor is null

Response Caching

List endpoints with filters return data that may not change between requests. Adding HTTP cache headers reduces unnecessary database queries and improves response times.

Cache-Control Header

The Cache-Control header tells clients and intermediate proxies how long a response can be reused:

Cache-Control: public, max-age=30

This says the response is cacheable by any cache (public) and valid for 30 seconds. For a parcel list, a short TTL balances freshness with performance.

ETag Header

An ETag is a fingerprint of the response content. The client sends it back on subsequent requests, and if the data has not changed, the server returns 304 Not Modified with no body.

Vary Header

The Vary header tells caches that the response depends on specific request headers. For filtered endpoints, the query string determines the response:

Vary: Accept, Accept-Encoding

Caching Strategy for Parcel Lists

Keep cache durations short for a parcel tracking system because clients expect near-real-time data.

Summary

In this lesson, you learned:

• Dynamic filtering with IQueryable builds SQL queries conditionally based on client parameters
• Cursor-based pagination provides consistent performance and stable results compared to offset pagination
• Sorting requires a tiebreaker field (usually the primary key) for deterministic ordering
• Page size limits protect the server from excessively large requests
• Pagination metadata (totalCount, nextCursor, hasNextPage) gives clients everything they need to navigate results
• Response caching with Cache-Control, ETag, and Vary headers reduces database load for list endpoints

Next, you will build dynamic query filters with IQueryable in ASP.NET Core.