TIL
Today I learned
Short notes on specific things worth remembering - a problem hit in production, the fix, and the one gotcha worth knowing before you copy it. 8 notes so far.
No notes match this tag.
Checksum the file before indexing it, not after
Two uploads of the same PDF shouldn't cost two embedding runs - hash the file up front and short-circuit indexing if that hash is already in the system.
A queue consumer needs to catch SIGTERM, not just crash on it
Kubernetes sends SIGTERM before killing a pod, and a queue consumer that ignores it drops whatever message it was mid-processing - catch it, finish the current message, then exit.
File-hash change detection keeps a big Lambda fleet's CI fast
Redeploying every Lambda function on every push doesn't scale past a couple dozen functions - hash each function's source and only redeploy the ones whose hash actually changed.
Keep agent code model-independent so a model swap doesn't change behaviour
Hardcoding a model's quirks into agent logic means every model upgrade risks silently changing behaviour - push model-specific details to config and keep the orchestration model-agnostic.
S3 presigned URLs get large uploads past API Gateway's payload limit
Amazon API Gateway enforces a 10 MB payload limit on REST APIs, and it isn't configurable - route the file straight to S3 with a presigned URL instead of proxying it through Lambda.
Token-aware batching beats fixed-size batching for embedding calls
Batching embedding requests by a fixed item count still hits 429s once individual chunks vary in length - batch by token budget instead, and back off on Retry-After.
A stale-job reaper stops a crashed worker from wedging a pipeline forever
A job marked 'processing' by a worker that then crashed or got OOM-killed stays 'processing' forever unless something else notices and resets it.
A headless-LibreOffice fallback for documents an AI service can't read
A document-intelligence service that reads PDFs cleanly can still choke on certain DOCX files - convert to PDF first with headless LibreOffice, then retry, instead of failing the upload outright.