REST-interview-quesitons.md

Senior Backend Engineer - REST API Interview Questions

1. Explain the Key Principles of REST Architecture

REST principles ensure that APIs are scalable and maintainable. Below are the key principles:

• Statelessness: Each request from a client to the server must contain all the information needed to understand and process the request. The server does not store any state between requests. This enables horizontal scaling since any server instance can handle a request without relying on others.

  • Example: In an e-commerce site, the session is not stored on the server, which means users can switch between pages seamlessly, and if the server restarts, the user's experience remains unaffected.

• Uniform Interface: REST relies on using standard HTTP methods (GET, POST, PUT, DELETE) and media types (such as JSON) to interact with resources in a predictable manner. This standardization reduces complexity.

  • Example: To fetch user details, you use a GET request to /users/123, whereas to delete a user, you use DELETE on /users/123.
  • HTTP Verbs Examples:
    • GET: Retrieve a resource. Example: GET /users/123 to fetch details of a user with ID 123.
    • POST: Create a new resource. Example: POST /users with a JSON payload containing user details to create a new user.
      • Request:

        POST /users
        Content-Type: application/json
        {
          "name": "John Doe",
          "email": "[email protected]"
        }

      • Response:

        HTTP/1.1 201 Created
        Location: /users/123
        {
          "id": 123,
          "name": "John Doe",
          "email": "[email protected]"
        }

    • PUT: Update an existing resource. Example: PUT /users/123 with a JSON payload to update the user with ID 123.
      • Request:

        PUT /users/123
        Content-Type: application/json
        {
          "name": "John Doe",
          "email": "[email protected]"
        }

      • Response:

        HTTP/1.1 200 OK
        {
          "id": 123,
          "name": "John Doe",
          "email": "[email protected]"
        }

    • PATCH: Partially update an existing resource. Example: PATCH /users/123 with a JSON payload to update specific fields of the user with ID 123.
      • Request:

        PATCH /users/123
        Content-Type: application/json
        {
          "email": "[email protected]"
        }

      • Response:

        HTTP/1.1 200 OK
        {
          "id": 123,
          "name": "John Doe",
          "email": "[email protected]"
        }

    • DELETE: Remove a resource. Example: DELETE /users/123 to delete the user with ID 123.
      • Request:

        DELETE /users/123

      • Response:

        HTTP/1.1 204 No Content

• Resource Identification: Resources are identified via URIs (Uniform Resource Identifiers). These resources are distinct, independent of the operations performed on them.

  • Example: /orders/456 represents an order resource, and interacting with this resource is always consistent.

• HATEOAS (Hypermedia as the Engine of Application State): Clients interact with a REST API entirely through hypermedia provided dynamically by the server (i.e., including hyperlinks for further actions within the responses).

  • Example: A GET request on /users/123 might contain links to view the user's orders or edit user details: { "orders": "/users/123/orders", "edit": "/users/123/edit" }.

• Cacheability: The server response must define whether the resource can be cached by clients or intermediaries. This helps reduce the number of interactions between the client and server, improving performance.

  • Example: A products list (/products) may include cache headers (Cache-Control) to indicate that this information can be cached for one hour, which reduces server load.

• Client-Server Separation: Clients and servers are separated, and each can evolve independently, as long as the interface is not altered. This encourages modular development.

  • Example: A mobile app (client) and a web dashboard (client) use the same RESTful backend API but are developed independently.

Trade-offs in Real-world Implementation: Strict adherence to REST principles can be limiting. For example, if a client needs data from several endpoints, it might require multiple HTTP requests. In such cases, a non-REST approach (e.g., GraphQL or custom endpoints that aggregate data) can reduce the number of network requests, balancing RESTful principles with practical performance needs.

2. Design a Versioning Strategy for a REST API

Versioning is crucial to maintain backward compatibility while evolving an API. Here is a strategy to manage versioning effectively:

• URL vs Accept Header Versioning:

  • URL Versioning (/v1/resource): Simple, easily understandable by clients. Useful for public APIs where backward compatibility is essential.
  • Accept Header Versioning (Accept: application/vnd.company.v1+json): A more elegant approach that uses content negotiation. It avoids polluting the URL but requires more sophisticated client handling.

• Breaking vs Non-breaking Changes:

  • Non-breaking Changes: Adding new fields or endpoints does not affect existing clients, and thus it’s preferred when extending the API.
  • Breaking Changes: Changes like modifying a response structure or removing fields are breaking. Introduce these in a new version to avoid breaking older clients.

• Documentation Approach: Use tools like Swagger (OpenAPI Specification) to document each version of the API. This allows consumers to understand version differences and makes integration easier.

• Client Migration Strategy: Deprecate older versions gradually by giving clients sufficient notice. Provide clear migration guides and deprecation warnings to enable them to upgrade without breaking their integrations.

• Version Sunset Policy: Set timelines for deprecating a version and communicate these to clients well in advance, such as "Version 1 will be supported until December 2025."

3. Implementing Rate Limiting for a Distributed REST API

Rate limiting is essential to protect an API from abuse or sudden spikes in requests:

• Token Bucket vs Sliding Window Algorithms:

  • Token Bucket: Clients receive a fixed number of tokens that refill at a regular rate. Useful for bursty traffic since clients can consume more tokens if available.
  • Sliding Window: Tracks requests in a moving window to determine whether a client is within rate limits. This approach is more predictable compared to the token bucket.

• Rate Limit Headers: Add headers like X-RateLimit-Limit and X-RateLimit-Remaining in responses to inform clients of their usage and remaining quota.

• Redis vs In-memory Implementation:

  • Redis: A distributed key-value store like Redis is suitable for large-scale distributed systems, where consistent rate limiting across nodes is required.
  • In-memory: For smaller setups or local environments, using in-memory rate limiting is simpler but limited to a single server.

• Multiple Rate Limit Tiers: Different users or roles may have different limits. For example, premium users may get a higher quota compared to free-tier users.

• Handling Burst Traffic: Token bucket helps handle short bursts while still enforcing limits over time.

• Client Identification: Identify clients using API keys or user tokens to apply rate limits accurately.

4. Securing a REST API

Securing an API is critical to avoid unauthorized access or data breaches. Below are common approaches:

• Authentication Mechanisms:

  • JWT (JSON Web Tokens): Tokens signed by the server and sent by clients in subsequent requests (Authorization: Bearer token). Suitable for stateless authentication. Common JWT payload includes:
    • User ID and roles for authorization
    • Token expiration time (exp)
    • Token issued time (iat)
    • Issuer information (iss)
    • Custom claims like subscription tier or permissions While session cookies are an alternative, they require server-side session storage which can be challenging to scale in distributed systems. JWTs are preferred here as they are self-contained and stateless, allowing any API server to validate the token without shared session storage.
  • OAuth2: Industry standard for delegated access. Provides scopes to limit user actions. Commonly used for public APIs.
  • API Keys: Useful for server-to-server communication.

• Authorization Patterns: Implement Role-Based Access Control (RBAC) or Attribute-Based Access Control (ABAC) to enforce permissions.

  • Example: Only an admin role can access /admin/users endpoint.

• CORS Policies: Cross-Origin Resource Sharing (CORS) restricts which domains can access your API. Set allowed origins and headers properly.

• Input Validation: Validate user input to prevent SQL Injection or Cross-site Scripting (XSS). Libraries like express-validator (Node.js) help enforce validation rules.

• API Gateway Integration: An API gateway (e.g., Kong, AWS API Gateway) acts as an entry point, providing additional security like rate limiting and request filtering.

• Security Headers: Use security headers (Strict-Transport-Security, Content-Security-Policy, etc.) to add layers of defense against common vulnerabilities.

• Rate Limiting and Throttling: Use rate limiting to protect against DoS attacks.

5. Designing a Caching Strategy for a REST API

To handle millions of requests efficiently, caching is a key tool:

• Cache Levels:

  • CDN (Content Delivery Network): Caches static content closer to users.
  • API Gateway Cache: Caches responses from the backend, reducing load. For example, using Nginx as an API gateway:

    # Enable caching in nginx.conf
    proxy_cache_path /tmp/nginx-cache levels=1:2 keys_zone=my_cache:10m max_size=10g inactive=60m use_temp_path=off;
    
    location /api/ {
        proxy_cache my_cache;
        proxy_cache_use_stale error timeout http_500 http_502 http_503 http_504;
        proxy_cache_valid 200 60m;    # Cache successful responses for 60 minutes
        proxy_cache_key $request_uri;  # Cache key based on full URI
        
        proxy_pass http://backend;
    }

    This configuration caches successful responses for 60 minutes and serves stale content if the backend is down.
  • Application Cache: Use in-memory cache (e.g., Redis) to cache frequent database queries.
  • Database Cache: Caching results at the database level using techniques like query caching. For example:

    -- MySQL query cache (before MySQL 8.0)
    SELECT SQL_CACHE * FROM users WHERE id = 123;
    
    -- PostgreSQL materialized view
    CREATE MATERIALIZED VIEW user_stats AS
    SELECT user_id, COUNT(*) as total_orders 
    FROM orders 
    GROUP BY user_id;

    The MySQL query cache would store the exact result for reuse, while PostgreSQL materialized views pre-compute and store complex query results.

• Cache Invalidation Approaches: Invalidate or update caches when the underlying data changes. Use a "write-through" pattern where changes are written to both the database and cache.

• Cache Headers and Directives: Use headers like Cache-Control, ETag, and Expires to define the cache policy.

• Cache Coherence in Distributed Systems: Use distributed caches and consistent hashing to avoid stale data and ensure even load distribution.

• Cache Warming Strategies: Preload popular data into the cache during off-peak times to avoid cold starts.

• Cache Hit Ratio Optimization: Continuously monitor the cache hit ratio with tools like Prometheus, and adjust caching strategies accordingly.

6. Pagination for Large Datasets

Pagination ensures scalability when dealing with large datasets:

• Cursor vs Offset Pagination:

  • Offset Pagination (?page=2&limit=20): Simple but has performance issues for large offsets. Consider a query:

    SELECT * FROM users ORDER BY created_at DESC LIMIT 20 OFFSET 980;

    This query must scan through 1000 rows to return just 20 rows, becoming increasingly slower as the offset grows. For page 50,000, it would need to scan 1,000,000 rows!

  • Cursor Pagination (?cursor=2023-10-01T15:30:00&limit=20): More efficient as it uses a reference point. Example query:

    SELECT * FROM users 
    WHERE created_at < '2023-10-01T15:30:00'
    ORDER BY created_at DESC 
    LIMIT 20;

    This query uses an index on created_at to jump directly to the right position, regardless of how deep into the dataset you are. The cursor is typically the last value from the previous page.

• Link Headers: Use link headers to indicate the next, previous, and last pages. This provides clients with easy navigation. Example response headers:

  Link: <https://api.example.com/users?page=3>; rel="next",
        <https://api.example.com/users?page=1>; rel="prev",
        <https://api.example.com/users?page=50>; rel="last",
        <https://api.example.com/users?page=1>; rel="first"

  This allows clients to traverse pages without having to construct URLs manually.

• Performance Implications: Offset pagination can be slow on large tables, especially without proper indexing. For example:

  Without index on created_at:

  SELECT * FROM users ORDER BY created_at LIMIT 20 OFFSET 10000;
  -- Full table scan required, very slow

  With index on created_at:

  -- Create index for better performance
  CREATE INDEX idx_users_created_at ON users(created_at);
  SELECT * FROM users ORDER BY created_at LIMIT 20 OFFSET 10000;
  -- Uses index, much faster

  Even better, using cursor-based pagination:

  SELECT * FROM users 
  WHERE created_at < '2023-10-01T15:30:00'
  ORDER BY created_at DESC 
  LIMIT 20;
  -- Most efficient approach

• Sorting Considerations: Ensure fields used for pagination are indexed to prevent performance degradation.

• Consistency Challenges: For frequently changing datasets, cursor-based pagination is preferred as it’s less affected by changes between pages.

7. Error Handling in REST APIs

A well-designed error-handling strategy is crucial:

• Standard HTTP Error Codes: Common HTTP status codes should be used appropriately:
  • 400 Bad Request: When request validation fails

    {
      "error": "Bad Request",
      "message": "Missing required field 'email'",
      "code": 400
    }

  • 401 Unauthorized: When authentication is required but missing/invalid

    {
      "error": "Unauthorized", 
      "message": "Invalid or expired access token",
      "code": 401
    }

  • 403 Forbidden: When user lacks permissions for the requested resource

    {
      "error": "Forbidden",
      "message": "You don't have permission to access this resource",
      "code": 403
    }

  • 404 Not Found: When requested resource doesn't exist

    {
      "error": "Not Found",
      "message": "User with id '123' not found",
      "code": 404
    }

  • 409 Conflict: When request conflicts with current state

    {
      "error": "Conflict",
      "message": "User with email '[email protected]' already exists",
      "code": 409
    }

  • 429 Too Many Requests: When rate limit is exceeded

    {
      "error": "Too Many Requests",
      "message": "Rate limit exceeded. Try again in 30 seconds",
      "code": 429
    }

  • 500 Internal Server Error: When server encounters an unexpected error

    {
      "error": "Internal Server Error",
      "message": "An unexpected error occurred",
      "code": 500
    }

  • 503 Service Unavailable: When service is temporarily unavailable

    {
      "error": "Service Unavailable",
      "message": "Service is under maintenance",
      "code": 503
    }

Custom application-level codes can be included alongside standard HTTP codes for more specific error handling:

• Error Response Structure: Provide a standardized response format for all errors, e.g., { "error": { "code": "VALIDATION_FAILED", "message": "Invalid email" } }.

• Validation Errors: Include error details specifying what fields failed and why. Example: { "errors": [ { "field": "email", "message": "Email is invalid" } ] }.

• Rate Limit Errors: Use 429 Too Many Requests when clients exceed rate limits, and provide headers to communicate retry intervals.

• Retry Strategies: Use exponential backoff to prevent overwhelming the server when retrying failed requests (500 errors).

• Error Documentation: Document all error codes, their meaning, and possible solutions for clients.

8. Designing a REST API for Collaborative Editing

Collaborative editing systems involve multiple users editing the same resource:

• Resource Modeling: Documents are primary resources (/documents/{docId}). Users interact by updating parts of a document.

• Concurrency Control: Implement optimistic locking by attaching a version number to documents. Clients send this version number when making changes, ensuring no conflicting modifications occur.

• Conflict Resolution: Notify clients about conflicting updates. For example, show differences in edited sections, allowing users to resolve manually (e.g., like Google Docs).

• WebSocket Integration: Use WebSockets to broadcast updates to all connected clients, ensuring real-time collaboration.

• State Synchronization: Use algorithms like Operational Transform (OT) or Conflict-Free Replicated Data Types (CRDT) to ensure consistent state between users.

• Event Notifications: Send events about changes, allowing clients to keep track of modifications.

9. Implementing Idempotency in REST APIs

Idempotency ensures that duplicate requests have the same effect as a single request:

• Non-Idempotent Example: POST requests creating new resources are typically not idempotent. Multiple identical requests create multiple resources:

  POST /orders
  {
    "product": "laptop",
    "quantity": 1
  }

  Each request creates a new order, resulting in:

  • First request: Order #1 created
  • Second identical request: Order #2 created
  • Third identical request: Order #3 created

• Idempotent Example: PUT requests updating a resource are idempotent. Multiple identical requests have the same effect as a single request:

  PUT /users/123
  {
    "name": "John Doe",
    "email": "[email protected]"
  }

  Multiple requests all result in the same final state:

  • First request: User 123 updated
  • Second identical request: No change, same state
  • Third identical request: No change, same state

• Idempotent HTTP Methods: Different HTTP methods have different idempotency characteristics:

  • GET: Idempotent - Reading a resource multiple times should return the same result
  • PUT: Idempotent - Multiple identical updates result in the same final state
  • DELETE: Idempotent - Deleting a resource multiple times has same effect as deleting once
  • HEAD: Idempotent - Like GET but returns only headers
  • OPTIONS: Idempotent - Returns supported HTTP methods for a resource
  • POST: Not idempotent - Multiple identical requests create multiple resources
  • PATCH: Not guaranteed to be idempotent - Depends on the update operation

• Best Practices:

  • Use PUT for full resource updates since it's idempotent
  • Prefer PATCH over PUT for partial updates, but ensure operations are idempotent

    • Bad Example: Non-idempotent PATCH that increments a counter (bad because each request changes state by incrementing, making it non-idempotent and unpredictable - multiple identical requests will keep incrementing the counter instead of resulting in the same final state)

      PATCH /products/123
      {
        "operation": "increment",
        "field": "views"
      }

      Multiple identical requests keep incrementing the counter

    • Good Example: Idempotent PATCH that sets specific values

      PATCH /products/123
      {
        "views": 42
      }

      Multiple identical requests all set views to 42, maintaining idempotency

  • Use POST when creating new resources where duplicate requests should create duplicates
  • Always use GET for retrieving resources, never for modifying state
  • Bad Practices and Solutions:

    • Exposing Database IDs: Directly exposing internal database IDs in URLs creates tight coupling

      # Bad - Exposes internal ID
      GET /users/42
      
      # Better - Use business identifiers
      GET /users/john.doe

    • Inconsistent Response Formats: Mixing response structures creates confusion

      # Bad - Inconsistent error formats
      GET /users/123
      { "error": "User not found" }
      
      GET /orders/456 
      { "status": "error", "message": "Order not found" }
      
      # Better - Standardized error format
      GET /users/123
      {
        "status": "error",
        "code": "NOT_FOUND",
        "message": "User not found",
        "timestamp": "2023-12-01T10:00:00Z"
      }

    • Verb-based URLs: Using verbs instead of nouns for resources

      # Bad - Verb-based endpoints
      POST /createUser
      GET /fetchOrders
      DELETE /removeProduct
      
      # Better - Resource-based endpoints
      POST /users
      GET /orders
      DELETE /products/123

    • Ignoring HTTP Status Codes: Using 200 OK for errors with error messages in body

      # Bad - Always returns 200
      GET /users/123
      Status: 200 OK
      {
        "success": false,
        "error": "User not found"
      }
      
      # Better - Use appropriate status codes
      GET /users/123
      Status: 404 Not Found
      {
        "message": "User with ID 123 not found"
      }

    • Not Versioning APIs: Making breaking changes without version control

      # Bad - No versioning
      GET /api/users
      
      # Better - Include version
      GET /api/v1/users

    • Deeply Nested URLs: Creating complex hierarchies that are hard to maintain

      # Bad - Deep nesting
      GET /users/123/orders/456/items/789/reviews/999
      
      # Better - Flatten resource hierarchy
      GET /users/123/orders
      GET /order-items/789
      GET /reviews?item=789

• Idempotency Tokens: When clients make a request, they include a unique idempotency key (Idempotency-Key: abc123). This key is stored by the server to prevent re-execution.

• Retry Handling: If the server receives the same idempotency key, it returns the previous response, ensuring that repeated requests don’t trigger the action again.

• Transaction Management: Ensure operations are atomic, completing fully or not at all, to prevent inconsistencies in distributed systems.

• Distributed Systems Challenges: Use a central store like Redis to store idempotency keys, ensuring consistency across distributed instances.

• Recovery Mechanisms: For failures during processing, log the progress, and roll back incomplete steps.

10. Testing REST APIs

Testing ensures reliability and quality of APIs:

• Unit vs Integration Testing: Write unit tests for isolated methods and integration tests for endpoint functionality.

• Contract Testing: Verify API contracts using tools like Pact to ensure compatibility between services.

  • Example: Consider a scenario where Service A (Consumer) depends on Service B (Provider):

    // Service A's Pact contract
    {
      "provider": "UserService",
      "consumer": "OrderService",
      "interactions": [{
        "description": "a request for user details",
        "request": {
          "method": "GET",
          "path": "/users/123"
        },
        "response": {
          "status": 200,
          "body": {
            "id": "123",
            "name": "John Doe",
            "email": "[email protected]"
          }
        }
      }]
    }

    This contract ensures that if Service B changes its API response structure, the Pact tests will fail, preventing breaking changes from reaching production.

• Performance Testing: Use tools like JMeter to perform load testing and determine the API's behavior under different conditions.

• Security Testing: Test for vulnerabilities such as SQL injection and XSS using tools like OWASP ZAP.

• Mocking Strategies: Use tools like WireMock, MockServer, Postman Mock Server, Nock (Node.js), VCR (Ruby), or responses (Python) to mock dependencies and third-party APIs, enabling isolated testing. Each tool offers different features:

  • WireMock: Java-based with REST API, good for microservices
  • MockServer: Supports multiple protocols including HTTP/HTTPS
  • Postman: Built into the Postman API platform
  • Nock: Popular for Node.js HTTP mocking
  • VCR: Records and replays HTTP interactions
  • responses: Lightweight Python library for mocking requests

• Test Data Management: Use Docker containers with seeded databases to replicate test environments consistently.

• CI/CD Integration: Integrate tests into the CI/CD pipeline (e.g., GitLab CI/CD) so they run automatically on every code push.

11. Handling Long-running Operations

Handling long-running processes involves:

• Asynchronous Processing: Process tasks in the background using a task queue like Celery (Python) to offload work from the request-response cycle.

• Status Polling: Allow clients to poll for updates. Example:

  # Server endpoint
  @app.route('/tasks/<task_id>/status')
  def get_task_status(task_id):
      task = Task.get(task_id)
      return {
          'status': task.status,
          'progress': task.progress
      }
  
  # Client code
  def poll_until_complete(task_id):
      while True:
          response = requests.get(f'/tasks/{task_id}/status')
          if response.json()['status'] == 'COMPLETED':
              break
          time.sleep(5)  # Poll every 5 seconds

• Webhooks: Register callback URLs to receive updates when tasks complete. Example:

  # Server side
  @app.route('/tasks', methods=['POST'])
  def create_task():
      task = Task.create(...)
      webhook_url = request.json['webhook_url']
      
      def process_task():
          # Do work...
          # When complete, call webhook
          requests.post(webhook_url, json={
              'task_id': task.id,
              'status': 'COMPLETED',
              'result': task.result
          })
      
      Thread(target=process_task).start()
      return {'task_id': task.id}
  
  # Client side
  @app.route('/webhook', methods=['POST'])
  def handle_webhook():
      task_result = request.json
      # Process completed task result
      return {'status': 'ok'}

• Progress Tracking: Include progress percentage or status updates (IN_PROGRESS, COMPLETED, etc.) in the status endpoint.

• Timeout Handling: Set a maximum time for a task to complete and handle failures gracefully if exceeded.

• Resource Cleanup: Ensure abandoned tasks are cleaned up, possibly using a scheduled cleanup process.

12. Handling File Uploads and Downloads

Efficient file handling in REST APIs is vital:

• Multipart Uploads: For uploading files, use multipart/form-data to enable multiple form fields, including the file.

• Chunked Transfer: For large files, split uploads into chunks, allowing clients to upload part by part.

• Resume Capability: Use byte-range headers (Content-Range) to allow interrupted uploads or downloads to resume.

• Progress Tracking: Use WebSockets to provide real-time updates to clients about the upload or download progress.

• Storage Integration: Use cloud storage services like AWS S3, integrating securely using signed URLs.

• Security Considerations: Validate file types, scan for malware, and use signed URLs to provide time-limited access to resources.

---

These answers provide a comprehensive understanding of RESTful API architecture and the challenges involved in building scalable, secure, and maintainable APIs. Let me know if you'd like any more details or examples from my experience! 🚀