Estimate project
← All posts

Data Subject Requests (DSRs) Explained, and How to Build a Scalable Solution for Them

The architectures presented here offer an efficient, scalable way of processing Data Subject Requests in a distributed system made up of many microservices.

7 min read
MicroservicesArchitectureSecurityDSR

A Data Subject Request (DSR) is a formal request that a person (the “data subject”) submits to an organization in order to exercise the rights granted to them by various privacy regulations. Such regulations, including the General Data Protection Regulation (GDPR) in the European Union and the California Consumer Privacy Act (CCPA) in the United States, give individuals particular rights over their personal data.

Types of DSRs

The most common kinds of DSRs are:

  • Right to Access: A request to see the personal data an organization holds.
  • Right to Rectification: A request to fix personal data that is inaccurate or incomplete.
  • Right to Erasure (Right to Be Forgotten): A request to have personal data deleted under certain conditions.
  • Right to Data Portability: A request to move personal data to a different service provider.
  • Right to Restrict Processing: A request to limit the ways in which personal data is processed.
  • Right to Object: A request to stop personal data from being processed for particular purposes, such as marketing.

Challenges of Handling DSRs in Distributed Systems

In systems composed of many microservices, organizations frequently struggle with handling DSRs. The main challenges are:

  • Data Distribution: Personal data is spread over many different microservices.
  • Inter-Service Dependencies: Completing a DSR may require services to perform operations in a particular order.
  • Scalability: The system must be able to process many DSRs at once without bottlenecks.

Architecture

Usually, DSR support isn’t a main concern when a microservices architecture is being designed – it doesn’t deliver core business value. Rather, it turns into a requirement that has to be met once the system is ready to process user data.

In our situation, the microservices architecture was already mature by the time we had to add DSR support. The proposed architecture relies on AWS Lambda for the processing logic, Kinesis Streams for communication, and microservices for carrying out particular data operations, which gives a scalable, modular, and auditable system for handling DSRs.

Key Components

  • API Gateway:
    • Serves as the entry point that receives DSR requests.
    • You can use it to expose an HTTP endpoint for your lambda, as described in the docs.
  • Orchestrator Lambda:
    • Checks the request and converts it into a task that task executors can consume.
  • Kinesis Stream for Task Propagation:
    • Works as a queue for the tasks produced by the Orchestrator Lambda.
    • Provides reliable, scalable delivery of tasks to the Task Executor Lambdas.
  • Task Executor Lambdas:
    • Several Task Executor Lambdas each take care of a different group of microservices, depending on the task type or area of interest, e.g.,: User Services, Commerce, Content, etc.
    • Every Task Executor Lambda reads tasks from the Kinesis Stream and calls its assigned microservices in the proper order, honoring dependencies.
  • Microservices:
    • Take care of particular parts of the DSR, e.g. retrieving, deleting, or updating data.
    • Every microservice emits events describing state changes, which can be used to track how the DSR is progressing.
  • Kinesis Events Streams:
    • All of our services were already emitting data-change events (much like CDC), which let us use those events to track DSR progress
    • Serve to track task status and to support asynchronous communication between components.
  • Completion Lambda:
    • Filters the events emitted by microservices to keep track of the DSR tasks’ status.
    • Combines the results and sends the final response.
  • Notifier (SNS/SES):
    • Notifies the requester about the DSR’s status or completion.
    • Any other notification mechanism would work too, but this is the easiest way to support application-to-person messaging.
  • Database/Storage:
    • Can optionally hold audit logs and track how DSRs are progressing.

Diagram of the Architecture

The diagram below may look complicated, but it simply reflects how challenging it is to implement a DSR solution for a system made up of dozens of microservices. It reuses the existing communication architecture, which propagates events through Kinesis streams. It could be described as a vertical architecture, in which execution flows from top to bottom.

vertical architecture

Workflow Description

  1. Request Initialization:
    • The API Gateway accepts the DSR request (e.g., a request to remove a particular user’s data).
    • The request is passed on to the Orchestrator Lambda, which validates it and creates tasks for the appropriate executors.
  2. Task Broadcasting:
    • The Orchestrator Lambda publishes tasks as events to a Kinesis Stream.
  3. Parallel Execution by Multiple Executors:
    • Several Task Executor Lambdas consume events from the Kinesis Stream. Each Task Executor Lambda is in charge of a particular group of microservices.
    • Inside each Task Executor Lambda, tasks run one after another when microservices depend on each other.
    • When any microservice fails to process the request, the entire task starts over from the beginning, which means every microservice has to support idempotency
  4. Status Tracking:
    • Every microservice publishes a data-change event back to a Kinesis Stream. This is part of the service’s regular behaviour and is unrelated to DSR handling.
    • The Completion Lambda watches the stream and combines the results, although picking out the right events can be tricky. One option is to add a dedicated field stating the reason for deletion, so that the Lambda processes only the relevant events.
  5. Final Response:
    • When every task is done, the Completion Lambda updates the status in the database and sends a notification through SNS/SES.
    • The requester receives the final response through the API Gateway.

Alternative approach

Another approach, similar to the one shown above, is also possible, particularly when your microservices architecture has no extra Kinesis streams for propagating events. In that case, the Executor lambdas can publish the results for each group of microservices to a dedicated Kinesis stream. This could be called a horizontal architecture.

horizontal architecture

The workflow is nearly the same as before, with the difference that execution progress is tracked through a single dedicated Kinesis stream rather than by gathering data from many streams.

Tracking DSR Progress

Key Principles

Keeping track of how Data Subject Requests (DSRs) progress is essential for maintaining visibility, meeting deadlines in line with compliance requirements, and keeping requesters informed about the status. Every step of the DSR process has to emit a status update describing its progress, which is stored in a central, queryable place.

Components for Progress Tracking

  • Kinesis Streams for Status Updates:
    • Every microservice publishes status updates (e.g., in-progress, completed, failed) to a dedicated Kinesis Stream.
    • The updates carry metadata like the DSR ID, task name, timestamp, and current status.
  • Completion Lambda for Aggregation:
    • Watches the status stream and combines the statuses for the DSR.
    • Keeps track of which tasks are pending, completed, or failed.
    • Writes progress to the central database in real time.
  • Centralized Database (DynamoDB/RDS):
    • Holds the progress and current state of every DSR.
    • Schema Example:
      • DSR ID: The DSR’s unique identifier.
      • Tasks: A list of every task together with its status.
      • Overall Status: Pending, In-Progress, or Completed.
      • Last Updated: The timestamp of the most recent update.
    • Makes it possible to query and audit the DSR process.
  • Notification Service (SNS/SES):
    • When enabled, informs requesters, for example:
      • When processing of the DSR starts.
      • When delays or problems occur.
      • When it is completed.
  • Dashboard (Optional):
    • A web-based or console interface that shows administrators the progress of every active DSR.
    • Offers search and filtering features.

Workflow for Progress Tracking

  1. Microservices Emit Status Updates:
    • Each microservice handles its task and publishes a status update to the Kinesis Data Stream when it completes, fails, or reaches an intermediate stage.
  2. Data Stream Feeds Completion Lambda:
    • The Completion Lambda reads updates from the Data Stream and writes them to the Centralized Database.
    • It decides whether every task has completed or whether errors have to be escalated.
  3. Real-Time Updates in Database:
    • The central database receives task-level progress for the DSR.

    • Example Record:

      {
      "DSR_ID": "12345",
      "Tasks": [
      {"TaskName": "Delete Data from Service A", "Status": "Completed"},
      {"TaskName": "Delete Data from Service B", "Status": "In-Progress"}
      ],
      "OverallStatus": "In-Progress",
      "LastUpdated": "2024-12-23T10:15:30Z"
      }
  4. Notifications and Dashboard:
    • Notifications fire when particular conditions are met (e.g., the DSR completes or is delayed).
    • Administrators follow DSR progress on a dashboard and step in when needed.

Best Practices for Progress Tracking

  • Consistent Metadata: Use a standard format for status updates so tracking stays uniform.
  • Retry Logic: Add retry mechanisms for failed tasks and log errors for auditing purposes.
  • Audit Logs: Keep a history of every status update for compliance and debugging.
  • Timeouts and Alerts: Define timeouts for tasks and raise alerts when deadlines are missed.
  • Privacy and Security: Make sure all tracking data is protected and complies with privacy regulations.

Advantages of the Mechanism

  • Real-Time Visibility: Both administrators and requesters can check DSR progress whenever they want.
  • Error Handling: Central tracking makes problems easier to spot and fix.
  • Compliance Assurance: Helps meet deadlines and produces a clear audit trail for regulatory needs.
  • Scalability: The system can process many DSRs at once without bottlenecks.

Advantages of the Architecture

  • Scalability:
    • Thanks to AWS Lambda, the system scales on its own to process many DSRs.
  • Modularity:
    • Each microservice takes care of specific tasks, so the system is easier to maintain and extend.
  • Dependency Management:
    • The Task Executor Lambdas make sure tasks run in the proper order, even when services depend on each other.
  • Audibility:
    • Logging and status tracking give a clear audit trail, supporting compliance with privacy regulations.
  • Asynchronous Communication:
    • Kinesis Streams let components communicate asynchronously, which lowers the risk of bottlenecks.

Conclusion

The architectures presented here offer an efficient, scalable way of processing Data Subject Requests in a distributed system made up of many microservices. Building on AWS services like Lambda and Kinesis Streams supports compliance with privacy regulations while keeping performance and reliability high. Tracking progress in real time improves visibility and helps organizations reliably meet regulatory deadlines.