Two slow endpoints, two fixes, and the number we do not have yet

The 26 September load test named two real bugs, and both are the kind that only show up when many people ask at once. The /api/latest/ endpoint returned a server error nine times in fifteen seconds under 50 simultaneous clients. The /api/ndvi-trends/ endpoint timed out on 50 of 150 requests, one in three.

A clue to the first came from asking the same endpoint three times. An endpoint that caches its answer gets faster each time: /api/regional/ dropped from 905 milliseconds to 108 and then 27. /api/latest/ barely moved, from 501 to 232 to 217, because nothing was caching its result. The chart at the end shows this for four endpoints.

On 27 September both were reworked. The latest-position endpoint now asks the database only for the ten columns it returns, keeps the answer for two seconds with a guard so many requests do not all rebuild it at once, and gains a new index through migration 0020. The commit also records that the index we first suspected already existed, so the lead in the original brief was wrong, and the fix was aimed at the real cost. The trends endpoint now asks the database one grouped question instead of one per country, caches the result for fifteen minutes behind a lock, rejects bad latitude, longitude or day values with a clear error, and reads only the validated Sentinel-2 rows, no longer a copy of MODIS data that had been reported as truth.

Here is what we have not done: rerun the load test after the change. The repository holds the before numbers and the tests of the new logic, and nothing else, so this post makes no claim about how much faster either endpoint now is. When the rerun exists, it will get its own post. The first test is described in the previous post, and the live service health is on the status page.

All of this is backend work for a pre-operations, simulation-only platform.

Read next

• One day, thirteen agents: the hardening day that came a few days earlier.
• Forty containers under load: the test that found these two bugs.
• Six commits to an MCP server: another 27 September build.

The detail chart

Latency of the first, second and third call to four endpoints, before the fix. Secondary to the story above.

Sources and credits

• Django cache framework: the caching tools behind the short-lived answers described above.