Skip to content

feat(serdes): Add filesystem serdes for large payload storage#494

Merged
ayushiahjolia merged 2 commits into
mainfrom
feat/filesystem-serdes
Jun 30, 2026
Merged

feat(serdes): Add filesystem serdes for large payload storage#494
ayushiahjolia merged 2 commits into
mainfrom
feat/filesystem-serdes

Conversation

@ayushiahjolia

Copy link
Copy Markdown
Contributor

Issue #, if available: #479

Description of changes:

  • Adds create_filesystem_serdes() - a SerDes implementation that stores serialized values on a durable filesystem (Amazon S3 Files or Amazon EFS mounted to Lambda). This keeps checkpoint sizes small regardless of payload
    size, with optional inline preview for observability.

Features -

  • ALWAYS mode: Every value written to file; checkpoint stores file pointer
  • OVERFLOW mode: Inline if < 256KB, overflow to file if larger
  • URI path encoding: Human-readable paths derived from ARN (<functionName>/<executionName>/<invocationId>/<operationId>.json)
  • HASH path encoding: Fixed-length SHA-256 paths for safety
  • Preview: Optional inline preview in checkpoint envelope for console/API visibility without reading the full file
  • ARN parsing: Compact directory structure from durable execution ARNs

Tests -

  • Added unit tests, e2e tests with replay and integration tests.

Next PRs -

  • Cloud integration tests (requires S3 Files or EFS mount infrastructure)
  • Documentation update in aws-durable-execution-docs repo

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

@ayushiahjolia ayushiahjolia force-pushed the feat/filesystem-serdes branch from 4bd0a83 to 84e6e0c Compare June 26, 2026 23:56
@ayushiahjolia ayushiahjolia marked this pull request as ready for review June 27, 2026 00:21
"create_filesystem_serdes",
]

logger = logging.getLogger(__name__)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this used?

and if not, how come ruff didn't notice, I wonder?

the package's pyproject.toml has [tool.ruff.lint] select = ["TID252"], does this means ruff is configured to enable only TID252 (the absolute-imports rule) and all default checks, including F401, are turned off?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not used and yes, select = ["TID252"] means ruff only enables that single rule. F401 (unused imports) and all other default checks are off. The CI's hatch fmt --check only enforces formatting (via ruff format) and that one lint rule.

I can followup with enabling F401 or all pyflakes checks if we want.

@ayushiahjolia ayushiahjolia force-pushed the feat/filesystem-serdes branch from acf1008 to 529ca4b Compare June 29, 2026 22:38
@ayushiahjolia ayushiahjolia requested a review from yaythomas June 29, 2026 22:45
@ayushiahjolia ayushiahjolia merged commit 9a35e55 into main Jun 30, 2026
732 of 823 checks passed
@ayushiahjolia ayushiahjolia deleted the feat/filesystem-serdes branch June 30, 2026 00:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants