ElasticBeanstalk HTTP Time

Web servers with NodeJS are fast, right? Yep, most of the time. The use case for our system is that users collect data with photos on their mobile app offline until they send their data. It's a nice hand-shake that the data exists on the device until the server receives it, so there is no data-loss and working offline was a requirement for the app. Typically an upload is a big JSON object and some photos. A few hundred K in most uploads. Recently one of our auditors keep a week's worth of data with a lot of photos, and gets a timeout when trying to send the data. 60 seconds and the upload fails. (We don't chunk it into multiple requests to maintain that handshake, and in 99.99% of the cases it is never a problem.)

We won't release a new mobile version for this single user, so the change needs to be made on the servers. Ours use ElasticBeanstalk, which is a fairly standard system setup, in that our traffic follows this flow:

LoadBalancer -> Instances, and each instance NGINX -> NodeJS

Lots of these stages can timeout on your HTTP requests, mostly by having a default. I have a testable HTTP request I made through Chome and watched the results in in the Dev Tools Network window. Initial testing died exactly at 60.

LoadBalancer

We are using AWS, so go to the EC2 service, on the left for Load Balancers, and change the "Idle Timeout." 

NGINX

This could have two places to change, one is NGINX itself (which is really handling all the web traffic) and the other is the NGINX-Node proxy. 

I made these changes in our server { context.

        client_max_body_size 50M;

        client_body_timeout 300s;

        keepalive_timeout 300s;

        send_timeout 300s;

And these in our location /api { (which is a proxy_pass to the nodejs upstream)

            proxy_send_timeout 300s;

            proxy_read_timeout 300s;

            send_timeout 300s;

These NGINX directives are all documented here: http://nginx.org/en/docs/dirindex.html. The main thing is to allow the connection to live with NGINX and upload for a while (the server changes), and that NGINX will keep the connection to node for longer (the location changes). These all had timeouts 30s - 60s.

This worked well! My test action made it to 120s. This increased my previous timeout values, but still not 300s like the settings suggested.

NodeJS

The default Node http servers could have timeouts on two places. First is the default server timeout (120s) and the second is the client request timeout. The server timeout defaults to 0 in Node 13 (and though we are using v12.18 this one was not an issue). Each request can have a different timeout as well. I made our two problematic endpoints have a longer timeout this:

req.socket.setTimeout(5 * 60 * 1000) // 5 minute timeout!

That was the final one. My test call actually completed in 4.3 minutes so the extended timings worked through the full system.

AWS SNS.publish Extended Message Data

Using AWS SNS in NodeJS to send Push Notifications, and we wanted to append extra data to the payloads. AWS has good docs for the plain text message, but trying to use a JSON Message object in the sns.publish call saw an error like:

016-07-12T16:19:09.713Z - error: InvalidParameterType: Expected params.Message to be a string

After some searching, I found the answer in an AWS forum dated back to 2013. Thought I would repost here for easier reference to myself.

The trick was those extra JSON.stringify calls, once to turn the platform messages into strings, and once to turn the entire Message object into a string. Seems like some odd parsing, but I have received notifications on an Android phone, so at least GCM likes it.

NodeJS on EC2 Instance

I needed a clean server to run a quick NodeJS server. We have been using Node for a while now and I haven't set up a server from scratch for a while. Either it's been working and we've just been using AMI images, or I used Elastic Beanstalk, which has done the Node setup already. There are some good blogs or tutorials on how to build node from scratch, but all the building is kind of tedious, especially if you picked a cheap server without all the processing power of the more expensive tiers.

So, to do this faster,

sudo yum update
sudo yum install openssl openssl-devel
sudo yum groupinstall "Development Tools"

That gets a few things ready on the system, and yum is quick. 

Next, I'm using Node Version Manager, or NVM for short. This is just a glorified bash script, though I use it on my dev machine. It's really great. (Especially if your last contract was still using Node 0.10 but pet projects want to use the latest stable version).  Also, it avoids all the source code building, which is the main point of this blog post : )

From the readme (all one line):

curl -o- https://raw.githubusercontent.com/creationix/nvm/v0.31.0/install.sh | bash

And that's it! I exited my ssh session, and came back. Then quickly ran

nvm install 5.9.1

Which worked flawlessly, and I could run node and npm in my terminal. 

$ node -v
v5.9.1
$ npm -v
3.7.3

I personally found that quicker than the other blogs out there. Hopefully it saves you some time as well.

Later Edit: When running after this install, I had an issue where my installed user could find Node (like above) but other users could not. The option I went with was from this blog, running the command:

n=$(which node);n=${n%/bin/node}; chmod -R 755 $n/bin/*; sudo cp -r $n/{bin,lib,share} /usr/local

And to quote the blog:

The above command is a bit complicated, but all it's doing is copying whatever version of node you have active via nvm into the /usr/local/ directory (where user installed global files should live on a linux VPS) and setting the permissions so that all users can access them.

This command would need to be run each time I ran a version. I also had to make sure /usr/local was in the $PATH of the users that needed Node.  And alternative to these extra commands would be to just use the nvm-global version of nvm (a different repo), which I might try next time.  But for now, I'm set up so will get back to it!

Parse to AWS - the Migration

We used Parse.com for a simple integration with Facebook login, and to manage users on a project. Parse was well documented and easy to work with. The integration took about a day, which was surely faster than managing our user accounts or writing our own Facebook authentication. The next week, Parse makes its announcement, which of course gave us some things to think about.

While our Parse usage was small and we had a few weeks before the data went lights-out, what would it take to transfer the service? The intent of this exercise is to determine what is needed and the steps necessary to transfer from real-Parse to an in-house version. The article by AWS made it seem easy, so I'll write here the things I found while reading there.

Server Setup

Planting Beans

AWS made it simple to create a new parse-server-example, with a one-click button to start a new application on their ElasticBeanstalk service. From start to finish it was about 4 clicks, but still really simple.

Note: You need AWS Policy access for this link to work. I added "ElasticBeanstalkFullAccess" to my IAM role group. Otherwise the link doesn't go anywhere useful, which you'll know because you can't select any options on the Beanstalk home screen.

Get Some Mongo

For Parse to work, you need a Mongo database (version 2.6.x or 3.0.x). You could roll your own on AWS, but having someone else do it for you is faster. We went the MongoLabs route for their free-tier. Signup and setup were as easy as expected.

Application Integration

And that about wraps up the AWS How-To. Now that we have a server, everything works right?

Well, not quite. Our app has three components that connect to Parse.com. A NodeJS server (that makes HTTP requests), a javascript web client and a Xamarin client (namely for iOS). Some things need to be addressed: urls and keys.

URLs

Most client SDKs point to "parse.com" or maybe "api.parse.com". How do we point these compiled client libraries to our new server?

Turns out, answers were at the bottom of the parse-server-example readme. There are values in the SDK libraries that allow the URL to be changed, and basically the key is not used on the self hosted parse server.

For our javascript SDK, which was running on SSL, putting in a non-secure URL didn't work, giving a good error like:

Mixed Content: The page at 'https://localhost:5010/game.html#/landing' was loaded over HTTPS, but requested an insecure XMLHttpRequest endpoint 'http://parseserver-xxxxx-env.us-east-1.elasticbeanstalk.com/parse/login'. This request has been blocked; the content must be served over HTTPS.

To turn on SSL on the beanstalk server, you need a certificate. AWS Certificate Manager can help sort those, and the Load Balancer configuration in the Beanstalk console can turn on port 443 listening using one of your valid certificates. 

Keys

The client applications are given keys by parse, typically for their own specific SDK. The "javascript key" allows javascript access on our web client, the "REST key" allows HTTP access using our server client, and same for Xamarin. How are those included in the Elastic Beanstalk instance, when the only environment variable is for the "Master Key"?

Also, the Facebook app ids need to live on the server somehow.

In the parse-server readme, there is the option to use environment variables which could be configured via the Beanstalk console. Whether the AWS server example respects them when the instances refresh remains to be seen.

Wrap Up

I'm stopping this investigation at the certificate chain since we don't have a wildcard cert, and SSL is needed in our app (so mixing non-SSL isn't an option for me). It would be possible to keep going, and I think the entire parse server would function as a drop in replacement with minimal code changes, but I'll have to revisit this at another time to completely integrate to our existing app.