Our project was due for an upgrade on Elastic Beanstalk. The platform running NodeJS v16 on AWS Linux 2 needed a blue/green deployment to reach v20. v18 had options for Linux 2 and Linux 2023, and v20 only Amazon Linux 2023 ("AL2023" from here on out).
Node v20 worked locally on all projects, time for cloud deployment.
There are documented differences between AL2 and AL2023, and even migrating AL2 to AL2023 on Beanstalk. Since we create PDF files using headless Chrome, I needed to update the packages for AL2023 (here, in my most popular gist). Deploy worked, beanstalk launched, good to go on vacation!
In fact, that's what I did, and it was during the 23hr drive home that I got "the call" stating there were some errors. Ugh.
The short version of the investigation was this: AL2023 on Beanstalk mounts /tmp in RAM whereas AL2's /tmp directory was on the main (EBS) disk.
Why is that important? If you recall those PDFs our system creates, it creates a lot of them. So much so that our 4GB RAM machines filled up their ~2GB /tmp storage directory. No more PDFs. This was never a problem on AL2 as /tmp was on the main disk, but was a complete surprise on the new instances of AL2023.
We fixed this by pointing our application's temp folder to a location on the disk, not the NodeJS default `os.tempdir()` which pointed to the /tmp directory in RAM. Now, like AL2, our temp files are stored on the main disk while using AL2023.
No comments:
Post a Comment