ElasticBeanstalk HTTP Time

Web servers with NodeJS are fast, right? Yep, most of the time. The use case for our system is that users collect data with photos on their mobile app offline until they send their data. It's a nice hand-shake that the data exists on the device until the server receives it, so there is no data-loss and working offline was a requirement for the app. Typically an upload is a big JSON object and some photos. A few hundred K in most uploads. Recently one of our auditors keep a week's worth of data with a lot of photos, and gets a timeout when trying to send the data. 60 seconds and the upload fails. (We don't chunk it into multiple requests to maintain that handshake, and in 99.99% of the cases it is never a problem.)

We won't release a new mobile version for this single user, so the change needs to be made on the servers. Ours use ElasticBeanstalk, which is a fairly standard system setup, in that our traffic follows this flow:

LoadBalancer -> Instances, and each instance NGINX -> NodeJS

Lots of these stages can timeout on your HTTP requests, mostly by having a default. I have a testable HTTP request I made through Chome and watched the results in in the Dev Tools Network window. Initial testing died exactly at 60.

LoadBalancer

We are using AWS, so go to the EC2 service, on the left for Load Balancers, and change the "Idle Timeout." 

NGINX

This could have two places to change, one is NGINX itself (which is really handling all the web traffic) and the other is the NGINX-Node proxy. 

I made these changes in our server { context.

        client_max_body_size 50M;

        client_body_timeout 300s;

        keepalive_timeout 300s;

        send_timeout 300s;

And these in our location /api { (which is a proxy_pass to the nodejs upstream)

            proxy_send_timeout 300s;

            proxy_read_timeout 300s;

            send_timeout 300s;

These NGINX directives are all documented here: http://nginx.org/en/docs/dirindex.html. The main thing is to allow the connection to live with NGINX and upload for a while (the server changes), and that NGINX will keep the connection to node for longer (the location changes). These all had timeouts 30s - 60s.

This worked well! My test action made it to 120s. This increased my previous timeout values, but still not 300s like the settings suggested.

NodeJS

The default Node http servers could have timeouts on two places. First is the default server timeout (120s) and the second is the client request timeout. The server timeout defaults to 0 in Node 13 (and though we are using v12.18 this one was not an issue). Each request can have a different timeout as well. I made our two problematic endpoints have a longer timeout this:

req.socket.setTimeout(5 * 60 * 1000) // 5 minute timeout!

That was the final one. My test call actually completed in 4.3 minutes so the extended timings worked through the full system.

AWS SNS.publish Extended Message Data

Using AWS SNS in NodeJS to send Push Notifications, and we wanted to append extra data to the payloads. AWS has good docs for the plain text message, but trying to use a JSON Message object in the sns.publish call saw an error like:

016-07-12T16:19:09.713Z - error: InvalidParameterType: Expected params.Message to be a string

After some searching, I found the answer in an AWS forum dated back to 2013. Thought I would repost here for easier reference to myself.

The trick was those extra JSON.stringify calls, once to turn the platform messages into strings, and once to turn the entire Message object into a string. Seems like some odd parsing, but I have received notifications on an Android phone, so at least GCM likes it.

Redis Ordered Set WITHSCORES into JSON Object (NodeJS)

A note for myself before I stop using this ordered set.

Redis Ordered Sets can return the members of a set with their scores. Its an option on the zrange command, so to get all the values in an ordered set with their scores:

`redis.client.zrange( key, 0, -1, 'WITHSCORES')`. 

Of course, the returned array is pretty specific, something like:

user1 135000 user2 384000 . . .

Using the handy lodash library, I wanted to keep track of this little hack:

//this is a synchronous operation and would block the event loopvar testScores = _.fromPairs(_.chunk(testValuePairs, 2));

I've seen it done with for-loops that skip every other array item, but I liked the single line of this approach. It does go through the array twice, once to turn into paired array, and once through that array to create the object, but for my purposes the array sizes were going to be small and the succinctness was useful.

NodeJS on EC2 Instance

I needed a clean server to run a quick NodeJS server. We have been using Node for a while now and I haven't set up a server from scratch for a while. Either it's been working and we've just been using AMI images, or I used Elastic Beanstalk, which has done the Node setup already. There are some good blogs or tutorials on how to build node from scratch, but all the building is kind of tedious, especially if you picked a cheap server without all the processing power of the more expensive tiers.

So, to do this faster,

sudo yum update
sudo yum install openssl openssl-devel
sudo yum groupinstall "Development Tools"

That gets a few things ready on the system, and yum is quick. 

Next, I'm using Node Version Manager, or NVM for short. This is just a glorified bash script, though I use it on my dev machine. It's really great. (Especially if your last contract was still using Node 0.10 but pet projects want to use the latest stable version).  Also, it avoids all the source code building, which is the main point of this blog post : )

From the readme (all one line):

curl -o- https://raw.githubusercontent.com/creationix/nvm/v0.31.0/install.sh | bash

And that's it! I exited my ssh session, and came back. Then quickly ran

nvm install 5.9.1

Which worked flawlessly, and I could run node and npm in my terminal. 

$ node -v
v5.9.1
$ npm -v
3.7.3

I personally found that quicker than the other blogs out there. Hopefully it saves you some time as well.

Later Edit: When running after this install, I had an issue where my installed user could find Node (like above) but other users could not. The option I went with was from this blog, running the command:

n=$(which node);n=${n%/bin/node}; chmod -R 755 $n/bin/*; sudo cp -r $n/{bin,lib,share} /usr/local

And to quote the blog:

The above command is a bit complicated, but all it's doing is copying whatever version of node you have active via nvm into the /usr/local/ directory (where user installed global files should live on a linux VPS) and setting the permissions so that all users can access them.

This command would need to be run each time I ran a version. I also had to make sure /usr/local was in the $PATH of the users that needed Node.  And alternative to these extra commands would be to just use the nvm-global version of nvm (a different repo), which I might try next time.  But for now, I'm set up so will get back to it!