Implement Pagination in Your FastAPI App

Master Pagination in FastAPI for Efficient Data Loading

As your application grows, serving all data at once can lead to performance issues and a poor user experience. This tutorial guides you through implementing backend-driven pagination in your FastAPI application, allowing users to load data in manageable chunks. You’ll learn to use query parameters, implement database pagination with SQL Alchemy, and integrate a ‘load more’ feature on the frontend.

Prerequisites

• A working FastAPI application (following previous tutorials in this series is recommended).
• Basic understanding of Python, FastAPI, SQL Alchemy, and HTML/JavaScript.
• The provided sample data generation script and images.

Step 1: Populate Your Database with Sample Data

To effectively test pagination, you need a substantial amount of data. The tutorial provides a populate_db.py script for this purpose.

1. Locate the script: Find populate_db.py in the tutorial files.
2. Review the script (Optional): This script creates sample users, assigns profile pictures, and generates 44 sample posts with realistic dates. It interacts with your API to create these records, ensuring they are processed correctly.
   • Warning: Running this script will clear your existing database and replace it with the sample data. Ensure this is intended before proceeding.
   • You will also need the images from the populate_images directory for profile pictures.
3. Run the script: Stop your FastAPI server if it’s running. Then, execute the script from your terminal:

   uvicorn run python populate_db.py

   (If you are not using Uvicorn, use your standard Python interpreter: python populate_db.py)

4. Verify data: After the script confirms the creation of users and posts, restart your FastAPI server (e.g., uvicorn run fastapi dev main.py) and check your application’s homepage. You should now see a significantly larger number of posts loaded at once, demonstrating the need for pagination.

Step 2: Define the Paginated Response Schema

The API needs a new contract with the frontend to communicate paginated data. This includes the data itself, along with metadata about the current page and total available data.

1. Open schemas.py: Navigate to your schemas.py file.
   • Tip: If your imports auto-sort, don’t worry if the order changes after saving.
2. Add the PaginatedPostResponse class: Append the following class definition to the end of the file:

   from typing import List
   from pydantic import BaseModel
   
   class PostResponse(BaseModel):
       # ... (existing fields for PostResponse)
       pass
   
   class PaginatedPostResponse(BaseModel):
       posts: List[PostResponse]
       total: int
       skip: int
       limit: int
       has_more: bool
   

Explanation of Fields:

• posts: A list containing the actual post data for the current page.
• total: The total number of posts available in the database.
• skip: The number of posts skipped to reach the current batch (offset).
• limit: The maximum number of posts requested for the current batch.
• has_more: A boolean indicating whether there are additional posts beyond the current batch. This simplifies frontend logic for displaying a “load more” button.

Step 3: Implement Pagination in the Get Posts Endpoint

Modify the existing API endpoint to accept pagination parameters and return the paginated response.

1. Open post router: Navigate to routers/post.py.
   • Add Query from fastapi and func from sqlalchemy to your imports.
   • Import the newly created PaginatedPostResponse from your schemas.
2. Update the get_posts endpoint function signature:
   • Change the response_model to PaginatedPostResponse.
   • Add skip: int = Query(ge=0, default=0) and limit: int = Query(ge=1, le=100, default=10) as parameters.

   Explanation of Query Parameters:

   • skip (ge=0, default=0): Ensures the offset is non-negative. Defaults to 0 if not provided.
   • limit (ge=1, le=100, default=10): Ensures at least one post is requested and caps the maximum at 100 to prevent resource exhaustion. Defaults to 10.
   • Tip: Using skip and limit offers more flexibility than page and per_page, aligning with common REST API practices.
3. Add a total count query: Before fetching posts, add a query to count all posts:

   
   total_posts = db.query(func.count(models.Post.id)).scalar()
   if total_posts is None:
       total_posts = 0
   

4. Modify the post fetching query:
   • Add .offset(skip) and .limit(limit) to your existing query.
   • Crucially, ensure an .order_by(models.Post.date_posted.desc()) clause is present. This guarantees consistent results across requests. Without ordering, the database might return different sets of posts for the same skip and limit values.
5. Calculate has_more: Determine if more posts exist after the current batch:

   
   has_more = (skip + len(posts)) < total_posts
   

6. Construct and return the PaginatedPostResponse: Instead of returning just the list of posts, create an instance of PaginatedPostResponse. You’ll need to manually serialize the posts using PostResponse.from_orm() (or similar Pydantic validation) to ensure nested data like the author is included correctly.

   
   return PaginatedPostResponse(
       posts=[PostResponse.from_orm(post) for post in posts],
       total=total_posts,
       skip=skip,
       limit=limit,
       has_more=has_more,
   )
   

Step 4: Test the Paginated API Endpoint

Use the FastAPI interactive documentation to verify your implementation.

1. Access the Docs: Ensure your server is running and navigate to your API’s documentation page (usually /docs).
2. Find the GET /posts endpoint: Expand the endpoint details.
3. Use “Try it out”:
   • Execute with default parameters (skip=0, limit=10). You should receive the first 10 posts, with total: 44, skip: 0, limit: 10, and has_more: true.
   • Experiment with different skip and limit values (e.g., skip=10, limit=10).
   • Test the edge case where you skip enough posts to exhaust the total (e.g., skip=40, limit=10). You should get the remaining posts, and has_more should be false.
   • Test validation by entering invalid values (e.g., limit=200). You should receive a validation error (HTTP 422).

Step 5: Implement Pagination on the Frontend (Homepage)

Update the main homepage route and template to fetch and display only the initial batch of posts, preparing for JavaScript-driven loading.

1. Centralize posts_per_page:
   • Create or update config.py.
   • Add a setting for the number of posts per page: POSTS_PER_PAGE = 10.
2. Update main.py (Home Route):
   • Import Query from fastapi and func from sqlalchemy.
   • Import settings from config.
   • Locate the home route function.
     • Replace the existing post fetching logic with pagination logic similar to the API endpoint, but without the skip parameter (as it’s always the first page). Use settings.POSTS_PER_PAGE for the limit.
     • Calculate has_more based on the number of posts returned and the total count.
     • Pass the has_more variable to the template context.
3. Add JavaScript Utilities:
   • Open static/utils.js.
     • Add an escapeHtml function to prevent XSS attacks by properly sanitizing user-generated content before injecting it via JavaScript.
     • Add a formatDate function to convert ISO date strings from the API into a more readable format (e.g., Month Day, Year) matching the server-rendered dates.
4. Update templates/home.html:
   • Wrap the existing post loop in a <div id="post-container"></div>. This div will be the target for appending new posts loaded via JavaScript.
   • Below the post-container div, add a button that will be conditionally rendered if has_more is true:

     
     {% if has_more %}
         <button id="load-more-btn">Load More Posts</button>
     {% endif %}
     

   • Add a <script> block at the end of the file.
     • Initialize state: Set initial values for currentOffset (equal to POSTS_PER_PAGE since the first batch is server-rendered) and hasMore using Jinja2 templating to inject server-rendered values.
     • createPostHtml(post) function: This function generates the HTML for a single post, mirroring the structure in the template. It uses escapeHtml and formatDate for security and display.
     • loadMorePosts() function:
       • Fetches data from the paginated API endpoint (/api/posts) using the currentOffset and POSTS_PER_PAGE.
       • Appends the newly fetched posts (converted to HTML using createPostHtml) to the #post-container.
       • Updates currentOffset.
       • Hides the “Load More” button if no more posts are available.
       • Handles errors by updating the button text to prompt retry.
     • Add an event listener to the #load-more-btn to call loadMorePosts() when clicked.

Step 6: Test Homepage Pagination

1. Reload your homepage. You should see the first 10 posts.
2. Click “Load More Posts”. New posts should dynamically load below the existing ones.
3. Continue clicking until all posts are loaded. The “Load More Posts” button should disappear.

Step 7: Apply Pagination to User Post Pages

Extend the pagination logic to individual user profile pages.

1. Update User Router (routers/user.py):
   • Add Query to FastAPI imports and PaginatedPostResponse from schemas.
   • Locate the get_user_posts endpoint.
     • Modify its signature to accept skip and limit query parameters, similar to the main get_posts endpoint.
     • Update the response_model to PaginatedPostResponse.
     • In the database queries (both count and fetch), add a filter .filter(models.Post.user_id == user_id) to retrieve only posts belonging to the specified user.
     • Implement the offset, limit, calculate has_more, and return the PaginatedPostResponse as done in Step 3.
2. Update Main Route (main.py):
   • Find the user_posts route function.
     • Add Query and settings imports.
     • Implement pagination logic using skip, settings.POSTS_PER_PAGE, and calculate has_more, similar to the homepage route.
     • Filter the post count and fetch queries by user_id.
     • Pass has_more to the user_posts.html template.
3. Update User Post Template (templates/user_posts.html):
   • This template is very similar to home.html. Wrap the post loop in a <div id="post-container"></div>.
   • Add the conditional “Load More Posts” button.
   • Add a <script> block at the end.
     • Initialize pagination state (currentOffset, hasMore).
     • The createPostHtml function will be the same.
     • The loadMorePosts function needs to be adjusted to fetch from the user-specific API endpoint (e.g., /api/users/{user_id}/posts) instead of the general /api/posts endpoint.

Step 8: Test User Post Pagination

Navigate to a user’s profile page. You should see their posts paginated, with a “Load More Posts” button if they have more than the limit.

Further Considerations

For more complex applications, consider using a dedicated library like FastAPI Pagination. While manual implementation is excellent for learning, libraries can automate much of the boilerplate code, offering various strategies and consistent response formats. Evaluate if the library’s benefits outweigh its potential complexity for your production environment.

Source: Python FastAPI Tutorial (Part 13): Pagination – Loading More Data with Query Parameters (YouTube)