Memory exhaustion from loading large datasets entirely into memory
warningResource ContentionUpdated May 21, 2024(via Exa)
Technologies:
How to detect:
Application loads entire database tables or large result sets into memory at once, causing memory exhaustion and potential out-of-memory errors. Occurs when using query.all() on large datasets.
Recommended action:
Replace query.all() with query.yield_per(1000) to process results as generators that yield records incrementally. For HTTP responses, use Flask's Response with a generator function to stream data without buffering entire result set in memory.