AWS Linux 2 -> AWS Linux 2023 temp directory

 Our project was due for an upgrade on Elastic Beanstalk. The platform running NodeJS v16 on AWS Linux 2 needed a blue/green deployment to reach v20. v18 had options for Linux 2 and Linux 2023, and v20 only Amazon Linux 2023 ("AL2023" from here on out).

Node v20 worked locally on all projects, time for cloud deployment.

There are documented differences between AL2 and AL2023, and even migrating AL2 to AL2023 on Beanstalk. Since we create PDF files using headless Chrome, I needed to update the packages for AL2023 (here, in my most popular gist). Deploy worked, beanstalk launched, good to go on vacation!

In fact, that's what I did, and it was during the 23hr drive home that I got "the call" stating there were some errors. Ugh. 

The short version of the investigation was this: AL2023 on Beanstalk mounts /tmp in RAM whereas AL2's /tmp directory was on the main (EBS) disk. 

Why is that important? If you recall those PDFs our system creates, it creates a lot of them. So much so that our 4GB RAM machines filled up their ~2GB /tmp storage directory. No more PDFs. This was never a problem on AL2 as /tmp was on the main disk, but was a complete surprise on the new instances of AL2023.

We fixed this by pointing our application's temp folder to a location on the disk, not the NodeJS default `os.tempdir()` which pointed to the /tmp directory in RAM. Now, like AL2, our temp files are stored on the main disk while using AL2023.

Actionhero Web Response Compression

Our company's recent internal tool was basically a database API with basic CRUD actions. There was some business logic to make sure things worked correctly, and I still like actionhero for my NodeJS APIs. The bulk of the project was in the web app (built with Quasar on top of Vue). I also wanted to try Docker and AWS Fargate, the instances are running actionhero directly (no nginx proxy).

Recently the company created a "big project". Most previous projects had 5-10 inner data hierarchies. This one had 206. The "/getFullProject" action had a real-world stress test! I thought VueJS would have issues, as in the past too many rendered objects or large arrays would slow down reactivity, but it works fine. Yay, version 3 : )

The hangup was the JSON download size. 120MB+ and it took over a minute. Too slow. Options are larger network tiers for Fargate, lazy-loading those data hierarchies from the web (API updates), or use nginx as a web proxy to gzip everything. However, I wanted a quicker solution and attempted to compress the JSON payloads directly from actionhero.

The compression project is meant for ExpressJS, but could we make it work in actionhero? Yes. Yes we can. (kind of : )

The result was the middleware above, that I only applied to a single action. My "/getFullProject" action. The `compression` project modified the raw request and response of actionhero (which are the base NodeJS http/https library). I had to turn off actionhero's normal response (data.toRender = false), just like building a file buffer or similar. Also the Content-Type header needed set before trying to compress; that is specific to the compression library (yay OpenSourceSoftware).  There are some issues with error handling, as modifying the `rawConnection` messes with how actionhero processes responses. And probably some other issues to be found! But it's an example of applying ExpressJS middleware inside actionhero.

For me, this reduced the 120MB+ payload down to under 8MB, and time from 1min+ to ~20s. All node changes, no API or infrastructure. 

It's still a slow action. I might change Docker to using a nginx proxy, and then lazy-load if needed. But for now, this experiment was sufficient and successful. 

Going with the Flow Interview Questions

I have done interviews that asked about what work I have done. Plus, of course, the overly technical interview. Two interviews really stood out as "good interviews" to me.  I actually enjoyed the interview questions! But the interviewer can tell a lot about how a person thinks and what they have really done. 

Both followed the same idea of a "going with the flow" style of interview question. Basically, the opening setting leads to a line of questioning that
1) is never wrong or discouraged
2) goes on for a while
3) requires the interviewer to think on their feet.

The point is dialog. How does the interviewee handle problem solving? Do they understand the concepts at large? The smaller parts too? Can they think on their feet? Have they done this before? And sometimes, do they give up on a challenge?

The two examples, paraphrased and reduced, went something like this:

Example 1:
Imagine a web page is up and a lawyer calls you at 3am saying it legally has to come down now. What next?
Me: Who is this? Are they for real?
Them: Always good to check! Yes, they are real.
Me: OK, I'd go turn off the server.
Them: It's on the cloud.
Me: Can I do it remotely?
Them: You could, but IT security just changed the console's access keys at midnight.
Me: Hmm, let's start at the front. I'd move the DNS to a different server.
Them: Good, but lawyer can't wait the hour for DNS propagation.
Me: Tune down at the load balancer / auto-scaling level so the server is turned off?
Them: Well, that could work, but your rules would apply to the other applications on the load.
Me: I might need to phone a friend. Who else is on my team?
Them: Teamwork, that's good. But they're all at a conference out of country, it's just you.
Them: Want to stop?
Me: Nah, let's keep trying. Can I SSH into the machine?
Them: Yes
....
Them: You don't have root access.
....

And on an on. The situation was not real, but it was real enough. The answers I gave might have worked or might not - I was never told. Their reasons for "not working" were never ending, and maybe did not make complete technical sense, but the exercise showed how far could I go? And many of these things are known from experience, not just reading blogs or doing starter projects.  The interviewer's final step when I ran out of ideas was to write a script to fill memory on the machine to starve the web server from running - I learned something! (. . . which, ironically, I ran into in our production environment a few years later - having no memory breaks stuff!)

Example 2:

Them: "We have a document processing system. Look at this chart. A goes into B, produces output 1, stored in C. D processes 1 into 2. E takes query criteria and runs against outputs 1 and 2.  We want to add process F, where would you start?"
Me: Let's add at new task at C.

Them: Cool. What technology would you use? 

Me: Looks like Redis is in A, and Redis is a good fit becase XYZ . . . 

Them: Alright, that task is working. What next?

Me: . . . .

Them: Aw, great idea! (even though it would never work. . . )

This one was more abstract because I forgot the details, and the details were really tied to their company use case. I could see the relation from the homework assignment I was assigned. Did I understand their company? Did I understand the tech being used? Was I a true "senior dev"? Every "idea" I had was a good one - or at least that's how the exercise was run. There was no need to go into the weeds, just "great idea, now what about this?!!" The interviewers were smooth too, so working with them sounded fun. They learned about me, I learned about them - win-win!

Wrap Up

This was a cool interview style that I'll keep in mind. For positions that require a lot of knowledge or experience, but you don't want to assign "build a full product" as homework, this type of questioning could be useful. Since it is very interactive, it can also pull out some personality that a coding exercise cannot.  It does require the interviewer to think on their feet and be knowledgeable of the context! Trying to fleece the interviewee could really backfire. 

Tips for Dev Experience

I was asked by a software developer recently how to grow in experience, in the context of job applications, and I put my thoughts here as an encouragement to all software developers out there looking grow in their career. 

Two memes (and there are many like each one)

Software technology changes quickly, each company has their own software “stack” and established companies rarely try all the new things. It would be impossible, or at least difficult, to be a pro at many programming languages and all the frameworks available. However, and thankfully, these days it is possible to gain experience outside the job. “Experience” can mean two different things:

1. General knowledge of how software stacks (purpose of each part of the stack, how pieces talk to each other, pros, cons, tradeoffs, etc)
2. Specific programming languages. 

The nice thing is, one can build up the other. We use a lot of JavaScript at work and I have a few years working with other languages (#2), but my knowledge of software systems (#1) is pretty good so even if a company used a language I am not most familiar with (say, Python), I could work on a little python project and the concepts I know (#1) will carry me enough on a weaker language (#2). 

So basically, a programmer these days can grow a personal portfolio of real projects until their work experience catches up. Build a real system using a database, server and web page (free). Host that system in the cloud– all have free tiers. Put your code in a code repository (GitHub, BitBucket – both free). There are many blogs out there that walk someone through each of those things. Same for building a mobile app, or sourcing your data from an open API or contribute to an Open Source project. 

Not only does this add a few lines on the ol’ resume, it is real experience building a software system and can be talked about during interviews. Opinions are formed. Dialog is easier. New concepts are understood easier. One can legitimately say, “I’ve done that” or “I’ve built that” and it’s not guessing or BS. If a real project has been built, the pieces need to work together. Best practices are found on-line. The bugs have to be investigated. It is running on the cloud. It is a real system! Promote that during interviews; “this is what I did,” and one can have the confidence to do so because the system was really built! And you'll have the passion to talk about it because it's "your" system (and us devs love our own code : ) 

A step further would be to do it again changing a few pieces of the system:

• use a NoSQL DB instead of SQL (or vice versa)

• host on a different cloud provider (AWS, Heroku, Google Cloud, MS Azure)

• make the same web pages with a different front-end framework (VueJS, AngularJS, ReactJS, etc)

• change the server (Node, Python, Go, Java, C#)

Same system with a different piece or two will give a lot of experience, and prepare anyone a little more to jump into a new team or software stack.

That’s my two cents. Certified education and academia has its place, and real work experience is key, but knowing how software works in practice carries a lot of weight as well, especially in early and mid career. This can be invested in with some off-hours time, and is available to anyone who is interested. 

Picking a project that is meaningful or personal helps a lot here. Some people just like to learn and do tutorials for fun. Others, like me, need a vested interest before spending the time. A potential business idea would be cool, but even a hobby interest can be enough to actually enjoy the work. And if one does not enjoy the software language, or the grind, or the learning, that is actually very valuable experience as well!

Git connections while using PdaNet+

Sometimes we travel to my parents' place, who do not have internet or wifi. 

~Me, basically

We do have cell service, so working remote is possible. I use PdaNet+, a nice utility app for Android that turns my phone into a wifi hotspot on the MacBook. This works great for most everything that uses https. (Special ports, like a direct connection to Postgresql 5432 and the like do not work, likely blocked by phone carrier). This Wifi Direct connection on the Mac (Big Sur) requires a special proxy port, but then email clients and web browsers work great. 

However, git was giving me problems. An error like this:

$ git pull

fatal: unable to access 'https://***@bitbucket.org/***/***.git/': Could not resolve host: bitbucket.org

All other http works, but this does not? Aaarrgh!!

The fix: Set the proxy directly in git as well (same that is needed in WiFi Network settings)

$ git config --add http.proxy http://192.168.49.1:8000

(don't forget to `git config --global --unset http.proxy` when done). 

Now git works too. I'm living large on the edge of civilization : )

Edit: When using SSH Git (like BitBucket or GitHub required in 2022) the config has to be set another way. These lines were needed in ~/.ssh/config (and needed to be removed/commented when done)

ProxyCommand nc -x 192.168.49.1:8000 %h %p

ProxyCommand nc -X connect -x 192.168.49.1:8000 %h %p

ElasticBeanstalk HTTP Time

Web servers with NodeJS are fast, right? Yep, most of the time. The use case for our system is that users collect data with photos on their mobile app offline until they send their data. It's a nice hand-shake that the data exists on the device until the server receives it, so there is no data-loss and working offline was a requirement for the app. Typically an upload is a big JSON object and some photos. A few hundred K in most uploads. Recently one of our auditors keep a week's worth of data with a lot of photos, and gets a timeout when trying to send the data. 60 seconds and the upload fails. (We don't chunk it into multiple requests to maintain that handshake, and in 99.99% of the cases it is never a problem.)

We won't release a new mobile version for this single user, so the change needs to be made on the servers. Ours use ElasticBeanstalk, which is a fairly standard system setup, in that our traffic follows this flow:

LoadBalancer -> Instances, and each instance NGINX -> NodeJS

Lots of these stages can timeout on your HTTP requests, mostly by having a default. I have a testable HTTP request I made through Chome and watched the results in in the Dev Tools Network window. Initial testing died exactly at 60.

LoadBalancer

We are using AWS, so go to the EC2 service, on the left for Load Balancers, and change the "Idle Timeout." 

NGINX

This could have two places to change, one is NGINX itself (which is really handling all the web traffic) and the other is the NGINX-Node proxy. 

I made these changes in our server { context.

        client_max_body_size 50M;

        client_body_timeout 300s;

        keepalive_timeout 300s;

        send_timeout 300s;

And these in our location /api { (which is a proxy_pass to the nodejs upstream)

            proxy_send_timeout 300s;

            proxy_read_timeout 300s;

            send_timeout 300s;

These NGINX directives are all documented here: http://nginx.org/en/docs/dirindex.html. The main thing is to allow the connection to live with NGINX and upload for a while (the server changes), and that NGINX will keep the connection to node for longer (the location changes). These all had timeouts 30s - 60s.

This worked well! My test action made it to 120s. This increased my previous timeout values, but still not 300s like the settings suggested.

NodeJS

The default Node http servers could have timeouts on two places. First is the default server timeout (120s) and the second is the client request timeout. The server timeout defaults to 0 in Node 13 (and though we are using v12.18 this one was not an issue). Each request can have a different timeout as well. I made our two problematic endpoints have a longer timeout this:

req.socket.setTimeout(5 * 60 * 1000) // 5 minute timeout!

That was the final one. My test call actually completed in 4.3 minutes so the extended timings worked through the full system.

When did I start my Mac

Sometimes I wonder when I started my work day, typically when I set up my laptop at the office. There is a command for this!

pmset -g log | grep 'Display is turned on' | tail

pmset -g log | grep 'Wake from Normal' | tail

You can try either one, and get some output like this:

2020-06-09 07:47:49 -0400 Notification         Display is turned on                      

2020-06-09 07:48:56 -0400 Notification         Display is turned on                      

2020-06-09 08:39:29 -0400 Notification         Display is turned on                      

2020-06-09 08:46:16 -0400 Notification         Display is turned on 

A short list of useful activity with the time of day.  Yay!