AWS Linux 2 -> AWS Linux 2023 temp directory

 Our project was due for an upgrade on Elastic Beanstalk. The platform running NodeJS v16 on AWS Linux 2 needed a blue/green deployment to reach v20. v18 had options for Linux 2 and Linux 2023, and v20 only Amazon Linux 2023 ("AL2023" from here on out).

Node v20 worked locally on all projects, time for cloud deployment.

There are documented differences between AL2 and AL2023, and even migrating AL2 to AL2023 on Beanstalk. Since we create PDF files using headless Chrome, I needed to update the packages for AL2023 (here, in my most popular gist). Deploy worked, beanstalk launched, good to go on vacation!

In fact, that's what I did, and it was during the 23hr drive home that I got "the call" stating there were some errors. Ugh. 

The short version of the investigation was this: AL2023 on Beanstalk mounts /tmp in RAM whereas AL2's /tmp directory was on the main (EBS) disk. 

Why is that important? If you recall those PDFs our system creates, it creates a lot of them. So much so that our 4GB RAM machines filled up their ~2GB /tmp storage directory. No more PDFs. This was never a problem on AL2 as /tmp was on the main disk, but was a complete surprise on the new instances of AL2023.

We fixed this by pointing our application's temp folder to a location on the disk, not the NodeJS default `os.tempdir()` which pointed to the /tmp directory in RAM. Now, like AL2, our temp files are stored on the main disk while using AL2023.

Actionhero Web Response Compression

Our company's recent internal tool was basically a database API with basic CRUD actions. There was some business logic to make sure things worked correctly, and I still like actionhero for my NodeJS APIs. The bulk of the project was in the web app (built with Quasar on top of Vue). I also wanted to try Docker and AWS Fargate, the instances are running actionhero directly (no nginx proxy).

Recently the company created a "big project". Most previous projects had 5-10 inner data hierarchies. This one had 206. The "/getFullProject" action had a real-world stress test! I thought VueJS would have issues, as in the past too many rendered objects or large arrays would slow down reactivity, but it works fine. Yay, version 3 : )

The hangup was the JSON download size. 120MB+ and it took over a minute. Too slow. Options are larger network tiers for Fargate, lazy-loading those data hierarchies from the web (API updates), or use nginx as a web proxy to gzip everything. However, I wanted a quicker solution and attempted to compress the JSON payloads directly from actionhero.

The compression project is meant for ExpressJS, but could we make it work in actionhero? Yes. Yes we can. (kind of : )

The result was the middleware above, that I only applied to a single action. My "/getFullProject" action. The `compression` project modified the raw request and response of actionhero (which are the base NodeJS http/https library). I had to turn off actionhero's normal response (data.toRender = false), just like building a file buffer or similar. Also the Content-Type header needed set before trying to compress; that is specific to the compression library (yay OpenSourceSoftware).  There are some issues with error handling, as modifying the `rawConnection` messes with how actionhero processes responses. And probably some other issues to be found! But it's an example of applying ExpressJS middleware inside actionhero.

For me, this reduced the 120MB+ payload down to under 8MB, and time from 1min+ to ~20s. All node changes, no API or infrastructure. 

It's still a slow action. I might change Docker to using a nginx proxy, and then lazy-load if needed. But for now, this experiment was sufficient and successful. 

Going with the Flow Interview Questions

I have done interviews that asked about what work I have done. Plus, of course, the overly technical interview. Two interviews really stood out as "good interviews" to me.  I actually enjoyed the interview questions! But the interviewer can tell a lot about how a person thinks and what they have really done. 

Both followed the same idea of a "going with the flow" style of interview question. Basically, the opening setting leads to a line of questioning that
1) is never wrong or discouraged
2) goes on for a while
3) requires the interviewer to think on their feet.

The point is dialog. How does the interviewee handle problem solving? Do they understand the concepts at large? The smaller parts too? Can they think on their feet? Have they done this before? And sometimes, do they give up on a challenge?

The two examples, paraphrased and reduced, went something like this:

Example 1:
Imagine a web page is up and a lawyer calls you at 3am saying it legally has to come down now. What next?
Me: Who is this? Are they for real?
Them: Always good to check! Yes, they are real.
Me: OK, I'd go turn off the server.
Them: It's on the cloud.
Me: Can I do it remotely?
Them: You could, but IT security just changed the console's access keys at midnight.
Me: Hmm, let's start at the front. I'd move the DNS to a different server.
Them: Good, but lawyer can't wait the hour for DNS propagation.
Me: Tune down at the load balancer / auto-scaling level so the server is turned off?
Them: Well, that could work, but your rules would apply to the other applications on the load.
Me: I might need to phone a friend. Who else is on my team?
Them: Teamwork, that's good. But they're all at a conference out of country, it's just you.
Them: Want to stop?
Me: Nah, let's keep trying. Can I SSH into the machine?
Them: Yes
....
Them: You don't have root access.
....

And on an on. The situation was not real, but it was real enough. The answers I gave might have worked or might not - I was never told. Their reasons for "not working" were never ending, and maybe did not make complete technical sense, but the exercise showed how far could I go? And many of these things are known from experience, not just reading blogs or doing starter projects.  The interviewer's final step when I ran out of ideas was to write a script to fill memory on the machine to starve the web server from running - I learned something! (. . . which, ironically, I ran into in our production environment a few years later - having no memory breaks stuff!)

Example 2:

Them: "We have a document processing system. Look at this chart. A goes into B, produces output 1, stored in C. D processes 1 into 2. E takes query criteria and runs against outputs 1 and 2.  We want to add process F, where would you start?"
Me: Let's add at new task at C.

Them: Cool. What technology would you use? 

Me: Looks like Redis is in A, and Redis is a good fit becase XYZ . . . 

Them: Alright, that task is working. What next?

Me: . . . .

Them: Aw, great idea! (even though it would never work. . . )

This one was more abstract because I forgot the details, and the details were really tied to their company use case. I could see the relation from the homework assignment I was assigned. Did I understand their company? Did I understand the tech being used? Was I a true "senior dev"? Every "idea" I had was a good one - or at least that's how the exercise was run. There was no need to go into the weeds, just "great idea, now what about this?!!" The interviewers were smooth too, so working with them sounded fun. They learned about me, I learned about them - win-win!

Wrap Up

This was a cool interview style that I'll keep in mind. For positions that require a lot of knowledge or experience, but you don't want to assign "build a full product" as homework, this type of questioning could be useful. Since it is very interactive, it can also pull out some personality that a coding exercise cannot.  It does require the interviewer to think on their feet and be knowledgeable of the context! Trying to fleece the interviewee could really backfire. 

Tips for Dev Experience

I was asked by a software developer recently how to grow in experience, in the context of job applications, and I put my thoughts here as an encouragement to all software developers out there looking grow in their career. 

Two memes (and there are many like each one)

Software technology changes quickly, each company has their own software “stack” and established companies rarely try all the new things. It would be impossible, or at least difficult, to be a pro at many programming languages and all the frameworks available. However, and thankfully, these days it is possible to gain experience outside the job. “Experience” can mean two different things:

1. General knowledge of how software stacks (purpose of each part of the stack, how pieces talk to each other, pros, cons, tradeoffs, etc)
2. Specific programming languages. 

The nice thing is, one can build up the other. We use a lot of JavaScript at work and I have a few years working with other languages (#2), but my knowledge of software systems (#1) is pretty good so even if a company used a language I am not most familiar with (say, Python), I could work on a little python project and the concepts I know (#1) will carry me enough on a weaker language (#2). 

So basically, a programmer these days can grow a personal portfolio of real projects until their work experience catches up. Build a real system using a database, server and web page (free). Host that system in the cloud– all have free tiers. Put your code in a code repository (GitHub, BitBucket – both free). There are many blogs out there that walk someone through each of those things. Same for building a mobile app, or sourcing your data from an open API or contribute to an Open Source project. 

Not only does this add a few lines on the ol’ resume, it is real experience building a software system and can be talked about during interviews. Opinions are formed. Dialog is easier. New concepts are understood easier. One can legitimately say, “I’ve done that” or “I’ve built that” and it’s not guessing or BS. If a real project has been built, the pieces need to work together. Best practices are found on-line. The bugs have to be investigated. It is running on the cloud. It is a real system! Promote that during interviews; “this is what I did,” and one can have the confidence to do so because the system was really built! And you'll have the passion to talk about it because it's "your" system (and us devs love our own code : ) 

A step further would be to do it again changing a few pieces of the system:

• use a NoSQL DB instead of SQL (or vice versa)

• host on a different cloud provider (AWS, Heroku, Google Cloud, MS Azure)

• make the same web pages with a different front-end framework (VueJS, AngularJS, ReactJS, etc)

• change the server (Node, Python, Go, Java, C#)

Same system with a different piece or two will give a lot of experience, and prepare anyone a little more to jump into a new team or software stack.

That’s my two cents. Certified education and academia has its place, and real work experience is key, but knowing how software works in practice carries a lot of weight as well, especially in early and mid career. This can be invested in with some off-hours time, and is available to anyone who is interested. 

Picking a project that is meaningful or personal helps a lot here. Some people just like to learn and do tutorials for fun. Others, like me, need a vested interest before spending the time. A potential business idea would be cool, but even a hobby interest can be enough to actually enjoy the work. And if one does not enjoy the software language, or the grind, or the learning, that is actually very valuable experience as well!

Git connections while using PdaNet+

Sometimes we travel to my parents' place, who do not have internet or wifi. 

~Me, basically

We do have cell service, so working remote is possible. I use PdaNet+, a nice utility app for Android that turns my phone into a wifi hotspot on the MacBook. This works great for most everything that uses https. (Special ports, like a direct connection to Postgresql 5432 and the like do not work, likely blocked by phone carrier). This Wifi Direct connection on the Mac (Big Sur) requires a special proxy port, but then email clients and web browsers work great. 

However, git was giving me problems. An error like this:

$ git pull

fatal: unable to access 'https://***@bitbucket.org/***/***.git/': Could not resolve host: bitbucket.org

All other http works, but this does not? Aaarrgh!!

The fix: Set the proxy directly in git as well (same that is needed in WiFi Network settings)

$ git config --add http.proxy http://192.168.49.1:8000

(don't forget to `git config --global --unset http.proxy` when done). 

Now git works too. I'm living large on the edge of civilization : )

Edit: When using SSH Git (like BitBucket or GitHub required in 2022) the config has to be set another way. These lines were needed in ~/.ssh/config (and needed to be removed/commented when done)

ProxyCommand nc -x 192.168.49.1:8000 %h %p

ProxyCommand nc -X connect -x 192.168.49.1:8000 %h %p

ElasticBeanstalk HTTP Time

Web servers with NodeJS are fast, right? Yep, most of the time. The use case for our system is that users collect data with photos on their mobile app offline until they send their data. It's a nice hand-shake that the data exists on the device until the server receives it, so there is no data-loss and working offline was a requirement for the app. Typically an upload is a big JSON object and some photos. A few hundred K in most uploads. Recently one of our auditors keep a week's worth of data with a lot of photos, and gets a timeout when trying to send the data. 60 seconds and the upload fails. (We don't chunk it into multiple requests to maintain that handshake, and in 99.99% of the cases it is never a problem.)

We won't release a new mobile version for this single user, so the change needs to be made on the servers. Ours use ElasticBeanstalk, which is a fairly standard system setup, in that our traffic follows this flow:

LoadBalancer -> Instances, and each instance NGINX -> NodeJS

Lots of these stages can timeout on your HTTP requests, mostly by having a default. I have a testable HTTP request I made through Chome and watched the results in in the Dev Tools Network window. Initial testing died exactly at 60.

LoadBalancer

We are using AWS, so go to the EC2 service, on the left for Load Balancers, and change the "Idle Timeout." 

NGINX

This could have two places to change, one is NGINX itself (which is really handling all the web traffic) and the other is the NGINX-Node proxy. 

I made these changes in our server { context.

        client_max_body_size 50M;

        client_body_timeout 300s;

        keepalive_timeout 300s;

        send_timeout 300s;

And these in our location /api { (which is a proxy_pass to the nodejs upstream)

            proxy_send_timeout 300s;

            proxy_read_timeout 300s;

            send_timeout 300s;

These NGINX directives are all documented here: http://nginx.org/en/docs/dirindex.html. The main thing is to allow the connection to live with NGINX and upload for a while (the server changes), and that NGINX will keep the connection to node for longer (the location changes). These all had timeouts 30s - 60s.

This worked well! My test action made it to 120s. This increased my previous timeout values, but still not 300s like the settings suggested.

NodeJS

The default Node http servers could have timeouts on two places. First is the default server timeout (120s) and the second is the client request timeout. The server timeout defaults to 0 in Node 13 (and though we are using v12.18 this one was not an issue). Each request can have a different timeout as well. I made our two problematic endpoints have a longer timeout this:

req.socket.setTimeout(5 * 60 * 1000) // 5 minute timeout!

That was the final one. My test call actually completed in 4.3 minutes so the extended timings worked through the full system.

When did I start my Mac

Sometimes I wonder when I started my work day, typically when I set up my laptop at the office. There is a command for this!

pmset -g log | grep 'Display is turned on' | tail

pmset -g log | grep 'Wake from Normal' | tail

You can try either one, and get some output like this:

2020-06-09 07:47:49 -0400 Notification         Display is turned on                      

2020-06-09 07:48:56 -0400 Notification         Display is turned on                      

2020-06-09 08:39:29 -0400 Notification         Display is turned on                      

2020-06-09 08:46:16 -0400 Notification         Display is turned on 

A short list of useful activity with the time of day.  Yay!

Integration Tests in Jest - Specific Group Serialization

Jest is a cool testing framework for NodeJS, and was adopted by my favorite API framework actionheroJS. In past projects I used Mocha, but to try something new I'm using Jest.

Background

I like to write integration tests when testing my APIs (I find them the most meaningful) which typically has a group of API calls in a single test suite. User Permissions, Client Configurations, Business Processes, etc are tested together.  Most times a folder contains the group of test for that suite, and tests are expected to run sequentially because the data created for the "create" api calls could be used for the "update" "read" and "destroy" API calls as well.

Enter Jest. Every test file is able to run in parallel and the state is independent of the other tests. Serial in a single file and parallel across many files. Great for running all the tests, but makes it hard to share state between them. To group my API calls I could either:

1. Put them all in a single file to test out the entire suite. Or...
2. Build up some testing data before each suite runs. 

I did not like #1 because the files would get huge. Each group might have 4-15 API calls that need exercised, plus setup/teardown and a few error cases. That's a lot of lines. Code folding in my IDE?

I did not like #2 too much either - custom testing code to "build the data" at the start of each test, plus a 5 sec or so boot time for a backend API instance to start on the server. Smaller files leads to a lot of boots and a lot of time (even with extra cores and Jest workers). Perhaps a DB seed file could ensure certain data would be available, but that's as dynamic as building on the fly and it's one more thing to maintain for the test suite.

What Does Not Work

To find a solution, I learned some things.

First, you cannot create a single, shared global to use across all your test suites. Lots of chatter about that here, but the summary is only JSON-serializable things can be global values. API tokens or WebSocket URLs? Sure. Instantiated objects with functions and all sorts of other initialized data, like an actionhero api instance?  No way. Each test suite is running in its own worker thread, meaning it has its own scope and cannot "talk to" the others (and don't try because they're all running parallel which could lead to craziness!).

Second, and this is true for many Node test frameworks (Mocha for sure), is that test setup (the describes) is synchronous while test running (it(), test(), beforeEach(), etc) is asynchronous. For Jest, you cannot instantiate a variable inside a describe() block that you want to use in your tests. You can initiate a variable, but must instantiate inside a test() or beforeAll() function.  The describe() functions all run at the start of testing synchronously, to setup what tests will run. The actual tests and variable instantiation happen when the test run, so if the variable was assigned in a describe() block, and not a test block, that assignment took place in the setup phase and likely does not contain what you expect it to (often leading to "cannot read property X of undefined" errors).

Solution

Let's jump into the solution, and it was as "simple" as sharing the API instance in the worker process scope in the same way that the api instance is used in the actionhero api.
Here we have a single actionhero instance started before all the test suites are run, and stopped after. Describe blocks are run in the order they are defined, so we describe the suites here, but the contents are in other files and can define their own beforeAll() or afterEach() functions. The convention is that are in their own describe() block.

Be sure to put that __test__/lib directory in your ignore patterns, or Jest will try to run them.

"jest": {

"testPathIgnorePatterns": [

"./__tests__/lib/*/*"

],

It is possible to change "describe('Create', createTests)" to "createTests()" by changing the export of the create.ts to be like this:

export default () => {

return describe('Create', () => {

I do not prefer it that way because some information would be hidden in 00-template-tests.ts, but it's an option!

You could probably do some fancy "walked directory" imports to find all the files in ABC order and include them in your suite - depending how large your suite grows or how annoying you find it to update the main test file.

That's how I'm running Jest tests in sequential groups, but maintaining parallel suites for all part of my API. Working for now, and I'll update this post if something goes terribly wrong : ) 

Winston v2 to v3 and Papertrail Logging

Winston is a logging library for NodeJS and has a bunch of community-built “transports” that use the Winston API to log to different places. Maybe “Console” and “Single File” while developing, and “logging service like papertrail” and “rotating log file” while in production. Code in the application “just logs” and configuration sends the messages where they should go. That how it works in our framework of choice, actionhero. 

Well, Winston went through a big upgrade to v3. This affected my project because the framework we were using had used Winston 2.x, and when I upgraded Node versions and our framework version, suddenly we were using Winston 3.x and my logs to papertrail stopped! We were using the Winston-papertrail transport which has been watching Winston v3 in a few different issues. While it worked, the logging looked junky because the transport was not updated and basically printed messages twice in the same log to Papertrail. Ugly, but at least functional again.

That’s where I ended yesterday and was just going to deal with it, when I found there was a v2 branch for Winston-papertrail. They say it’s not complete (some connect error cases) and I had to change my formatting a little in my application, but it works lovely and everything is back and running again. The community is slow moving on the Winston-papertrail project but it’s hanging in there and any Node project moving from v2 to v3 of Winston and wants Papertrail should be aware of it.

The Senior

Titles are a funny thing. Sometimes we like to play them down, and sometimes we like to have big, fancy ones. Banks typically make lots of people "V.P."s so you know you're talking to somebody important. The CEO of my last company said, "you could call me the janitor as long as I got paid more than you." When it comes to developer positions, an often used term is "Senior" and it seems to mean different things at different places.

What is "The Senior"? 

At a previous company, a large defense contractor, there was a technical progression for Software Engineers: Associate -> Engineer -> Senior -> Staff -> Senior Staff -> Principle. It went on from there but not many reached the higher ones. (Remember that previous CEO of mine? ; ) Being a "senior" at that company put you squarely in at a Level 3 Technical position, and was directly related to your experience, education and pay scale. Outside of these extremely large organizations, a "senior developer" means many different things. And it is not consistent among companies.

Sometimes the title is self-defined. If someone thinks they are "senior" and can convince their boss or hiring manager the same, they'll get the title. Some companies have (loosely or strictly) defined criteria for what senior means, and some are too small to think about titles so it seems like people just make things up. I'm looking at you, "Chief Awesomeness Officer at New Startup Company." Everyone there gets a good title because both of you needed cool business cards : )

Is a senior based on experience in years? Skill? A gut feeling - "I know it when I see it." I've seen the title claimed by people that were skilled and/or experienced, but still felt something was lacking. They were good at what they did, but were they a senior in my eyes? Something seemed off, the title did not match.

Trust

To wrap up, for organizations without a defined progression structure, it comes down to trust. What can I trust this person with? 

• new team member/mentee? 
• entire team?
• research project or incubator idea?
• maintain a legacy project that business critical? 
• product roadmap?
• deadline?
• customer (interaction)?

Some people might have skills at some of these, and some experience as well. Ultimately it comes back to the stakeholders, and if they can trust their "Senior" with their product. Placing people in their strength areas is important, and in my opinion, a senior could simply be trusted with more. Maybe more in a few select areas, or just more areas in general. 

In the end, do not worry about titles. Keep working hard and growing your skill set so that you can be trusted with more. A good indication of what you could do is what you have already done (e.g. - What have you done in production setting that earned money?) 

Stakeholders will notice, and are seeking to find, people they can trust with their product. 

Performance Matters

Performance matters. And when we develop code we want it to be lean and efficient. Those algorithm courses we took in college and the study of Big O notation surely lets us write performant software. Algorithms that are fast, using the minimal amount of memory are sure to wow our stakeholders!

[Those of you working in the embedded space can probably skip this entire post. Performance does matter where CPU, GPU and RAM are limited. I cut my teeth for years programming towards and recording Technical Performance Measures on avionics display software. Every millisecond counted and when our load time requirement of "3 seconds" failed because of "3012 ms" - anything is on the table to increase performance. If performance is the requirement, it matters everywhere, and you should care all the time! ]

When does performance matter?

Unless performance is part of your requirements, it doesn't.

Most game and app developers, or those using modern web technologies (front end or server-side), well, performance does not matter that much.

At least, it does not matter that much right now.

When developing a modern system, do not focus on the optimal performance of your code in the first pass. Yes, you might be able to hand code a more efficient sorter than .sort() and you might save a few bytes from your network packets if you minify/uglify your payloads yourself. You might even move an algorithm from O(n^2) to O(n log n) in only a few hours.

The thing is, it just does not matter right now. Until your code is deployable in production, time spent optimizing and tweaking for the sake of algorithm performance is wasted time. Get it working. Make sure it is robust, passes your tests and can be deployed. That is the only time to go back and optimize. Focus on writing readable, maintainable, testable, deployable code first.

Does performance ever matter

For most projects, it probably doesn't. It could be a good exercise to code some whiz-bang algorithm or try to reduce the number of statements or calls needed in a function. Once your software is working and deployed is the time to know if performance matters. Why spend time optimizing something that might change? Or a feature that gets cut?

Consider these things as you decide if might matter:

Human-speed vs computer speed

Computers are really fast. Humans are not. If you are optimizing for something that will be noticed at human speed, don't. Why worry about a list of 100, 1000, or 10,000 items? You CPU will burn through those operations whether sorting or mapping or whatever. If the function you are performing is part of a user interaction, they will not notice the time savings between 157ms and 194ms. And a computer will do a lot in those 37ms. 

How often does this happen

Some functions and data flows happen often and concurrently. Speeding up those could have noticeable gains, even to us slow humans. However, some flows happen very irregularly. We have a special admin action that will gather a certain report. The report is costly to run, but is only requested maybe weekly. If all our concurrent users asked for this report every few seconds, our system would come to a halt! Time to optimize! As it is, the time needed to optimize is not worth the single, transient blip. Let's spend our effort on more worthwhile User Stories. 

Just scale up

Hardware is cheap. Can your current performance metric just be solved by buying a bigger machine? If your DB layer is running slow, is it worth your time to profile and optimize all your SQL statements and maybe rearrange your data tables? Or could you just buy a bigger server? Same for CPU, RAM, Network, etc. Do not sweat these performance decisions too early (after your initial planning) because sometimes a bigger machine will fix a lot of problems. In today's cloud world, a push of a button and little more yearly expense could save hundreds of hours of Dev/QA time. 

How to know if performance matters

Ultimately, you'll never know if performance matters if you are not measuring it. Performance metrics are absolutely necessary to 1) know where to focus and 2) know if it worked! Anyone that tells you "that's slow" or code reviews "that's not efficient" has absolutely no data to back that up! What is "slow"? What is "efficient"? Is there a suggestion that will satisfy this arbitrary feeling of slow? Until you can measure performance, you cannot know what is not performant or if you got better!

For a modern web system, you will see performance losses at the system (and typically network) level more than your code. Making 10 HTTP calls in series? You might notice that. Consider how those can be reduced or run in parallel. Is your DB more efficient with multiple JOINs or separate queries? What user actions are used the most? What are the slow ones? What is most efficient to make stakeholders happy? 

Start measuring and measure as much as you can. Then you'll really know where performance matters.

Using Puppeteer in AWS Node ElasticBeanstalk

Our company project has to send out reports and I like the flow of generating those reports from a web page to PDF conversion. We can reuse the tools and skills we use on our web analytics dashboards to develop the reports, with the added benefit that the reports look exactly like the web pages.

I first saw this flow from PhantomJS, the scriptable, headless browser. Load a web page into Phantom, say "gimme a PDF of that web page", re-configure settings 50 times to get it to look better, and boom!, you have a PDF of that web page. This is what we were using in our NodeJS server until reading this note and running into this issue. So we switched to Puppeteer, a similar project that was made for NodeJS and backed up by Google Chrome team.

Our servers deploy into AWS ElasticBeanstalk, their Node Environment. We haven't the need for containers or ECS yet, though did need a trick to get Puppeteer working in project. Things worked locally, but certain required things were not on the AWS Linux machines we were using, and getting those libraries installed on Beanstalk was a bit of a hassle.

A lot came from this StackOverflow question/answer (which itself came from somewhere else). Ultimately, through some trial-and-error, I created an ebextension config file to allow Chromium to install when Puppeteer is installed for the Node project.

File is below, and locally at .ebextensions/chromiumpackages.config. These run in order, first the yum packages and then the mysterious rpm commands. Note the use of --replacepkgs, otherwise your script will run the first time and fail subsequent times because the packages are already there. Yes, I guess technically we are downloading files and try/failing rather than checking if they exist, but it sure does keep the file simple!

And that's it. This config file runs when beanstalk makes a new instance, and then Chromium installs cleanly when my NodeJS project installs puppeteer. Yay!

NGINX HTTPS Redirect on AWS Elastic Beanstalk

We run a NodeJS app on AWS Elastic Beanstalk. Their node environment works well for us and on the machine is NGINX running on port 8080, and our node app running at port 8081. Some proxy table forwards port 80 to 8080, so NGINX receives all the traffic.

The SSL certificate is managed by AWS Certificate Manager, and is configured at the Load Balancer. That load balancer is serving traffic on port 80 (HTTP) and 443 (HTTPS), but always to port 80 on the beanstalk instance(s). It's very good to not have to worry about SSL in NGINX or node, and I could access our pages via HTTP or HTTPS.

However, I always wanted to use HTTPS. How could I automatically forward? Changing the Load Balancer to different ports (like 443 -> 80 to 443 -> 443) would have required some deep system changes on the beanstalk instances, which is handled by some complex beanstalk scripts. Examples are out there, but it all seemed overly complex and then I was tying our system to some random internet script that I did not write or understand.

Thankfully the solution was much simpler.

This is in my NGINX setup:

  server {
    listen 8080;

    if ($http_x_forwarded_proto = "http") {
      return 301 https://$host$request_uri;
    }

    // lots of other stuff
  }

That's all it took, and I understand it! If the protocol is "http", redirect for "https" and keep everything else (host and URI) the same. Mostly this will only affect the first load of the page, after that all future requests will remain HTTPS anyway.

(Only downside is sometimes I forget the "s" when pasting a `curl` command, and the response doesn't follow the redirect, responding with 301.)

SSH EC2 to EC2 and Security Groups

I ran into the case where my shiny new Elastic Beanstalk instances wanted to talk to some older services running on standard EC2 instances, and the method to do that was via SSH.

Problem

I use SSH often to manage that older service and connect with an Elastic IP (which is basically a static IP) as the "hostname". Trying this same approach worked in dev (our office has a static IP and it's just like my terminal), but failed when deployed to Elastic Beanstalk.

Managing the Security Group (SG) of the older service required a new access rule for port 22:
Using the SG of Elastic Beanstalk failed to connect.
Using the private IP of of our VPC (e.g. 172.31.0.0) failed to connect.
Using the public IP of the Elastic Beanstalk worked!

However, this was problematic because the IP of my elastic beanstalk could change (our staging system is a single instance but production is a rolling cluster). Editing the SG manually would be dumb and writing a script to check public IPs in the Beanstalk after each deploy sounded hard.

Easy Solution

Instead of referencing the static IP address XXX.XX.XXX.XXX when connecting via SSH, I used the public DNS, which contains the elastic/static IP address anyway (e.g. ec2-XXX-XX-XXX-XXX.compute-1.amazonaws.com, so shouldn't change on me).  This allowed EC2 to resolve internal IP addresses, and thus the Security Group rule on the older EC2 instanced worked for another security group instead of public IP. 

I panicked and asked on AWS Forums and StackoverFlow as well, and answered my own question at each. 

Why we chose actionhero?

While there are many frameworks for NodeJS servers, I have been using a framework for building our applications for the past few years and wanted to sum up those thoughts here.

TL:DR

actionhero

Production API Server

I've read a little about Jade and EJS as rendering engines for Node servers, namely Express, but never wanted to jump into having my front-end and back-end running on the same server. I had brief thoughts on the performance implications of having my single-threaded Node process serving and rendering web pages (over nginx or other proxies), and there were other reasons along those lines, though the scale of these projects were not going to be web-huge anyway. Mainly, I just did not want to tie my web page's HTML with server models, forever locking both together. (Having inherited a Java/JSP system, pulling the two apart in any language would leave some scars!)

Actionhero is almost purely for an API. Yes, it can serve static files, but to me it was built for data processing and message passing. I had flexibility in my client technologies because they simply spoke to an API. There are many features that are production ready and tested. This framework can be used for small prototypes to full-scale production systems.

Organization

My first Node project was built on top of Restify which was a fine choice for our data system (different team was building the front-end, another benefit of an API server). The concepts of Restify were solid, but I ended up developing my own system of organizing all my scripts. While that taught me a lot about Node's `require()` and the loading order of a Node app, it was a pain! Making sure all my end-points got loaded with clever directory layouts. Time spent on avoiding mixing the response handling and data retrieval. In the end it worked but there was time lost building, essentially, a framework for our app from scratch!

Actionhero has a folder structure for components of the system: actions, initializers, configs, and tasks. Each component type has a purpose, code layout and sometimes a load order. There is a flow to how the server works - where "actions" are the focus. Starting a new project with actionhero, or jumping into another actionhero project, has everything laid out the same way and I do not need to worry about it. Another win! Personal favorite concept of organizing code with actionhero is thin Actions and fat Initializers - the concept works so well in practice.

Features Included

Many of the other web server frameworks pride themselves in being "light" (Restify, Express) which is good, and then I'm spending lots of time choosing logging frameworks, middleware, websockets, config environments, scaling, etc. Flexibility is great, but the outcome of my choice was probably not vital, plus would have cost the time of research and integration.

Actionhero has some opinions and I'm content at the extent they are chosen. When a html app needed events pushed to it, actionhero already had a chat system and WebSockets. When a client did not want WebSockets in their Unity3D app, actionhero already handled every HTTP action via a TCP server. When I needed a dev, test and production environment for our system, a simple config controlled each one. A large list of common features to any production Node system are "built in", and while certain pieces are very opinionated, there is a lot of flexibility for everything else.

Documentation and Community

Final thing worth mentioning were the docs, which I've linked to a few times already. They are well written and comprehensive . . . most any answer I looked for could be found there first. And since just enought is included with the framework, and my code stays organized, I find myself writing more useful code than figuring out how to write the code.

Also, the community of actionhero is top-notch. This is due to actionhero's creator and top-contributor, Evan Tahler, who runs a great project. It's tested, documented, and evolving - and has been the past few years. The Slack channel is particularly useful these days, even over Stackoverflow or Github issues.

Shortcomings

Just to list these, these are my two short comings with actionhero worth mentioning.

1. Redis is required. At least to utilize the chat or scalability (different server-nodes communicating) features. While a caching database isn't a terrible idea anyway, it's one more piece in the cloud infrastructure (usually one of more costly pieces) and there is not a choice if you'll only use some other similar DB.  For me, Redis is awesome anyway and until you need nodes to scale, it's not even necessary for a single-server-node (or server-node independent system).
2. Ecosystem is small. When compared to Express, probably the most popular Node web server framework, there are far fewer plugins, blogs, examples, stackoverflow questions, and everything else for actionhero. Just not as many people use it, though due to documentation and community I have not found this to be a problem. Also, due to code organization, I know how to wire some npm package into the framework, and can use that package's documentation to figure out any issue there.

If this is your first node project ever, follow along a blog for Express to make a HTTP endpoint respond "Hello World." If you want to make a full-sized web application server, and be ready for all those production issues and feature growth you aren't even thinking about right now, get started with actionhero. That's my go-to choice while working with NodeJS.

Upgrading postgresql tools on Amazon Linux

Tools like pg_dump, pg_restore and psql.

I didn't find much help when trying to do this, so thought I would write it up here.

Our AWS RDS was Postgres 9.6.1 but the postgres tools on the EC2 instance from the default yum repos was 9.5.4. Can't `pg_dump` with the minor version mis-match!

These were my commands (lines starting with // are like commands, don't run those : )

// Start at yum.postgresql.org 
// https://yum.postgresql.org/repopackages.php#pg96
// Reading through docs there, need epel repos enabled

sudo yum-config-manager --enable epel

// Now there are more postgresql packages listed, but not 9.6!

yum list postgresql*

// Need to install the matching rpm file with yum. Copy the link.

wget https://download.postgresql.org/pub/repos/yum/9.6/redhat/rhel-6-x86_64/pgdg-ami201503-96-9.6-2.noarch.rpm

sudo yum install pgdg-ami201503-96-9.6-2.noarch.rpm 

// Yep, can see the packages for 9.6

yum list postgresql*

sudo yum install postgresql96.x86_64

// there was an error. Uninstall the 9.2 package (how did that even get there?)

sudo yum erase postgresql92

sudo yum install postgresql96.x86_64

// All tools now report 9.6.1

pg_dump --version

pg_restore --version

psql --version