Upgrading postgresql tools on Amazon Linux

Tools like pg_dump, pg_restore and psql.

I didn't find much help when trying to do this, so thought I would write it up here.

Our AWS RDS was Postgres 9.6.1 but the postgres tools on the EC2 instance from the default yum repos was 9.5.4. Can't `pg_dump` with the minor version mis-match!

These were my commands (lines starting with // are like commands, don't run those : )

// Start at yum.postgresql.org 
// https://yum.postgresql.org/repopackages.php#pg96
// Reading through docs there, need epel repos enabled

sudo yum-config-manager --enable epel

// Now there are more postgresql packages listed, but not 9.6!

yum list postgresql*

// Need to install the matching rpm file with yum. Copy the link.

wget https://download.postgresql.org/pub/repos/yum/9.6/redhat/rhel-6-x86_64/pgdg-ami201503-96-9.6-2.noarch.rpm

sudo yum install pgdg-ami201503-96-9.6-2.noarch.rpm 

// Yep, can see the packages for 9.6

yum list postgresql*

sudo yum install postgresql96.x86_64

// there was an error. Uninstall the 9.2 package (how did that even get there?)

sudo yum erase postgresql92

sudo yum install postgresql96.x86_64

// All tools now report 9.6.1

pg_dump --version

pg_restore --version

psql --version

Everyone Has Good Ideas

Everyone has good ideas. At least someone thought enough to give the idea. And while some ideas are really not that good in the end, you do not know until you hear them. When building a software product, ideas come from everywhere.

The business-side will have some ideas and you know they will be good ones, because they are paying for everything! Amazing how ideas sound a little different from the person (with)holding the checkbook. Developers will have good ideas because they are creative and just finished reading medium.com, and are eager to try new framework/library/database because surely it will be amazing and solve all the things. Other people in the company, or even individual clients, will have ideas now and again, sometimes in a very niche part of the software. And those ideas are important to them, more important than your next major or minor release!

What to do?!! Blowing people off and just blazing ahead with your plan might turn out well, and you'll prove to everyone how right you were! But at best some people are not happy, with you or your processes. At worst, you stop receiving new ideas at all. (Let's not even consider ignoring others ideas and still sinking . . . uh oh for you!)

When it comes to receiving ideas, two things need to happen:

People need to feel heard
People need feedback

Feeling Heard

Like I mentioned before, any idea someone brings up is important to them. If they do not feel heard or valued when giving their idea, they might become alienated to the team or just bitter. Could happen differently for different people, but the basic principle is to treat people well! We want good ideas, and we want people to give them.

Feeling heard is just a quick discussion, and it is also helpful in clarifying the idea.

• "What were you thinking?" 
• "Why is this important?" 
• "Any timeframe the idea?" 
• Maybe there's a quick drawing on a whiteboard. 

If you can repeat back the idea after you understand it, that person will feel valued.

All this will go into your backlog of choice, whether its a spreadsheet, sticky notes in your dev's cube, or some tool that team uses. Ready to evaluate and enter planning (or cold storage : )

If you hear people's ideas and they have no way to get any feedback, their ideas just go into a blackhole and they have no clue if the idea is something to prepare for. We contracted on a team once where the Tech Lead wanted the business team to leave the dev team alone, constructed a wall, and ideas were tossed over (through him). Without any feedback from their meetings, the sales team were pitching one set of ideas to clients, the dev team was building another set of ideas, and everyone only found out how far apart they were at Milestone Releases! Contrast that with a company I've spoken with that has great intra-team communications, and ideas are flowing!

Giving Feedback

Often called closing-the-loop, any idea needs a resolution.

• Is it happening this sprint? 
• Is it up for future review? 
• "Yes, though wait until Milestone 2 is over." 
• Skip it, because...

It does not matter whether this feedback is personally communicated or can be looked up at another time. It does matter why the resolution turned out the way it did.

People need to know what was good about the idea, and what was not. Maybe the idea was good, just not right now. Maybe the impact would not be worth the investment. Whatever the reason, it communicates the goals and schedules of the team, and will help shape better ideas in the future.

If people's ideas get feedback, but they are not heard, the people do not feel valued. Ever hear things like this?

• "Why bother giving ideas, they just get rejected?" 
• "Those tech guys don't know anything, this idea is necessary and will save the company!" 
• "Yeah, I saw the demo. That was my idea, ya know?"

People that feel valued will contribute more, will stay with the company, will think of real ways to improve because they genuinely care. Good people are the greatest asset, and must be valued (of course, a principle in life and not just business.)

Now Do It

The process your team implements can be adjusted, but both parts are necessary.

The existing meetings (Planning, Retros, "weekly team", etc) you already have are a good place to touch on these things. Keep using whatever tool you're using to manage tickets. With an attentive team lead (scrum master, whatever), even "being heard" can be accomplished remotely through the ticketing system; though starting that way may not have the right affect until people can trust the process. Even give rewards for good ideas. Whatever it takes for your specific team.

Everyone has good ideas, so work hard to find and keep the good ones, and ensure the next ones are brought in.

The Overly Technical Interview

Recently I started the interview process with a tech company that worked on some pretty cool software. That was their business, they recently went public, and were growing quickly. I applied as a Technical Lead (forget the actual title) and my background was a fit to get past the screening interview.

The Test

First step past the screening interview was a timed test at HackerRank. 75 minutes to answer 15 multiple choice and 3 programming problems. The multiple choice were a range of topics:

• What CSS style would be applied to this element?
• What is the correct SQL query for generating this table output?
• What would this code output on the console?
• Where is the programming mistake in this (pseudo) code?
• Which algorithm is more efficient?

A great range of questions and topics. The only bad thing about multiple choice is second guessing yourself, like knowing the answer in 5 seconds and then talking yourself out of it for two minutes : )

The programming problems were pretty cool, and the testing tool allowed you to write and run in a large variety of languages. It even unit tested for you! My first question I got to a valid place to test and passed 2 of 5 unit tests, but didn't get to see the inputs or expected outputs of the tests. So I rewrote it a different way, and this time got 3 of 5 of the unit tests. And by this point time was running out, so I moved on.

In the end I ran out of time before completing every question. I did not see immediate feedback on my test (from the tool or company's recruiter) and was not sure how this would be scored. Was I intended to finish? Was this some kind of stress test? Do they only take those who can finish?

The One-On-One

I guess my score was enough to move on and I had a one-on-one with another engineer. We talked about past projects of mine, and then did a coding exercise together, live on codeshare.io. The problem was to write Fibonacci in any language I wanted. I just started coding, though it did not feel natural because I haven't used the Fibonacci sequence since I learned about recursion in CS 101. I took a break from coding to explain where my head was at and explained a looping algorithm that would solve the problem by keeping track of an index and two variables. My interviewer showed a recursive sequence and asked it's pros and cons (algorithm complexity, stack size, etc) from the version I described. I asked about his views on the company and we were out of time.

The End

The next day I got the "thanks but no thanks" response from my recruiter, which of course bubbles up the feelings of not getting picked. Everyone was nice and professional, and we're off our separate ways, wishing each other the best.

Takeaways

This interview did not lead to an offer, which is the most telling thing that can come from an interview process. In my view, something did not go right because I thought an offer was possible. It is why I applied!

Two takeaways from this experience. One for me, one for them.

Takeaway One: Study

For those tech companies that are asking the algorithm questions, and I've heard about it from a lot of the big names, one needs to study. No matter if you think the questions are "too basic" or "just facts you can look up on-line", this is the screening process. If you want to stand a chance with these types of companies, study! Know the tricks. Eat Fibonacci for breakfast, B-Trees for lunch and some Big O notation for dinner. Practice at HackerRank or some other coding problem site.

Also, I would have used algorithm design comments more than actual code, especially when I was going to run out of time in the test, or during the coding interview where code would not run. Partial credit in those situations would hopefully have shown I knew what needed to be done ("solved the problem") and just needed time to code it correctly.

In the end, this is the company's interview process and if I want to compete in that space, I need to hit the books!

Takeaway Two: Interview for the Position

I mentioned this was for a Technical Lead position, yet feel that every multiple choice question and especially my one-on-one problem solving interview, could have been solved by looking up StackOverflow or Wikipedia. Not a single question about managing a project timeline, following a software process, thoughts on testing, system design, choosing a framework or component architecture. Maybe there would have been more of that past this "screening technical interview", but if I were a team lead, one would think those items would be important.

Same goes for the timed test and code-sharing, to a degree. Was that an accurate measure of my technical knowledge? No time for design, because the test clock was running and we had 9 minutes left on our interview when we hit codeshare.io. HackerRank is a cool tool, but is coding without (more than one) test inputs or expected outputs very common? Speed coding might be fitting at a hack-a-thon, but "just start typing" is not a typical way to get the best results at the office. The concern is while these tests/reviews might be good for screening, is it relevant to your actual work?

Other interviews I have participated in required a homework assignment with a relevant problem to what the position was for. Time was not stressed. Code was not forced/constrained by a tool. We could discuss the work result, and how I got there. To me, though it is more time invested, feels more relevant to the job at hand. Perhaps that is seen more with smaller companies, but the big ones that are hiring 2-20 people a month, let alone screening 5x-10x that, just don't have the bandwidth for more. Though I wonder if turnover has any correlation to the interview process chosen.

Interviewing is its own set of skills, and I'd better stay sharp : )

Fast and Slow Questions

I like going to art galleries and am genuinely impressed by at the skill and creativity of the artists showing their works. I'm typically drawn to the more technical works that, in my mind, highlight ability over expression. Photorealistic pieces are my favorite. Sometimes I like the abstract or minimalist, but that's just less often. Everyone has different tastes, because we are all a little different, and that's great!

The one thing about art galleries is that it does not take me more than five seconds to know if I like a piece or not. I do not need to, or care to, stand and ponder. When it comes to engineering projects, there are fast questions and there are slow questions. Certain times or certain clients might require one over another, and the questions you are asking will drive the outcome.

Slow Questions

A typical slow question might be "What do you want?" or "What do you need?" Generally this could be called brainstorming, and whether that session takes an hour, or even more sessions, the question is slow. The feedback takes times to think about. The big picture is not fully formed. Everyone has different ideas and goals in mind.

Sometimes slow questions are needed. Requirements need to be generated. Technologies and processes need to be evaluated. Requirements need to be updated. 3rd Parties need consulted or included. Maybe it becomes painfully slow! Yet by the end, you have a well designed system and everyone is on the same page with the outcome expectations.

Fast Questions

Fast questions have fast answers. "Yes or No?" "What don't you like?" The fast question will demand fast feedback, and is often iterative. This is not a single question and then everything is done, but one question leads to the next iteration and the next fast question, yet things keep moving and people remain engaged.

Experience plays a huge factor in being able to ask a fast question. Knowing what is out there, and how to apply tools or technologies. Knowing what has worked and what has not. Proof-of-concepts with a demo project. This type of experience allows for questions that can be answered and acted upon quickly.

What Questions Are You Asking?

If you work with the DoD or for a large enterprise, often you'll be asked (or asking) slow questions. A lot of people, risk mitigation and contract preparation needs to be set up prior to the work. It can feel slow, yet often is necessary.

Smaller teams and those trying to get an edge (first to market or finding new business) cannot afford the slow questions. Either a client is not willing to pay for the perceived delay or they do not have the answers to your slow question anyway ("What do you need?"). Having a prototype could be key for faster situations, because it allows for a fast response. "Is this what you meant?" "What changes before we go live?" These are fast questions and can accelerate development and interest.

The types of questions we ask are the types of answers we will receive. At the art gallery, I'll still ask the fast ones.

The Rush, in Retrospect

Preparing for the meeting did not go well. Live, on-site, with client and end customer, and it could have been better.

Crud.

The actual demo was a success, but getting there was not. What happened was not fun. It was not fun for the client, whose expectations were not met. Not fun for our bosses, who dealt with the brunt of the fallout and had reputations on the line. Not fun for the dev team, who worked hard to get the demo even to the state it was in, plus felt the guilt for a less-than-stellar technology rollout.

Now that the smoke has cleared, the yelling and finger-waving has stopped, and blame has been passed around, we can take a look at what happened. Two questions need to be asked:

How did we get here?
How do we keep from happening again?

Getting here was the easy part. Because everything is fine, before it is not! Looking with hindsight, there were a few things that changed silently. Dates got pushed up a few weeks. Demo changed from a tech team demo to a customer meeting. Meeting turned from a demo into a pre-launch. Feature or two changed from the plan. That little technical hurdle that takes longer than planned.

Thinking everything was on track was easy, but as changes to expectations started trickling in from different places, they were not all in sync when the meeting rolled around. And that is how we got here.

The thing is, by then it was too late. There was not time for fallback plan. Strength of will or more management oversight cannot make anything happen faster, or even just fast enough. The yelling was endured, the plan was laid out, the weekend was worked and with a final push, the launch was successful.

Hooray?

The result of a successful launch is good, but getting there smoothly is a better goal. Since this was not smooth, how can we keep it from happening again? Ultimately it falls on us. The client will always be the client, so they may miss art delivery deadlines by weeks, set meetings without our input, and always ask more (for less). Might be annoying at best and infuriating at worst, but clients will be clients!

The point of process is for situations like these, when expectations are key. There were no defined requirements or checklists of what was needed at what date, just that "it should work and be awesome." Somewhere our software process broke down, which led to bad expectations. Running with a broken process is overly misleading too, so by not assessing things early we did not address issues soon enough. At each change in dates, we needed to have been very clear about what features were going to be ready, not a mix of "everything in the contract, just less, with different dates."

The other thing that would help things run smoother is a defined contract style. In broad terms, the types of contracts we have accepted are fixed price and retainer. Quick recap of each:

Fixed Price

• Defined set of features to deliver.
• Defined delivery dates.
• Set price

Retainer

• No specific features, but team is available to work
• Dates are defined for retaining purposes ("start of each month")
• Price is "per time" (hours worked, or monthly)

Fixed price is good when someone knows what they want, and requires a lot more up front planning. People always want fixed price (and want it to be lower!), but often do not take the time to define features. Good for fixed deadlines (trade show demo, bottom-line price for release), and often has some contention toward the end like "that feature was expected" "it obviously should work this way".

Retainer is a good model when people know they need work done, but cannot define what that is, or have a tendency to change their minds, or need to see it to know if it was "right." Usually things are smooth in the start and middle of a contract, and the problems come up at the end of a retainer, where people always question the effort without knowing how long it should have taken in the first place.

Back to the post at hand, we mixed the two. We were signed up for a fixed price with fixed deadlines, then changed the deadlines and were working like a retainer model. Most any change that was requested was started to be incorporated, which was OK until some people were thinking the original deadlines ("no problem, plenty of time!") and others were thinking the (new) demo dates. Now doing retainer-style work was not meeting specific and defined constraints, so while everyone was "busy", it was not focused on the important things.

The contract is now basically concluded, with time to spare! Learning from ourselves this time, we'll likely improve for the next. 

Prototype Rocks Are Better

A common analogy with my bosses is "Bring me a rock." This fictional conversation between an "idea person" and "engineer" would go something like:

Idea Person: "Bring me a rock."
Engineer: "Here is a rock."
IP: "It is not dark enough."
E: "Here is another rock."
IP: "Why is it so jagged?"
E: "You want a darker, smooth-ish rock?"
IP: "Yes, obviously."
E: "Here is a rock."
IP: "...it's too small."

We are not geologists, and this conversation is not really about rocks.

Ideas are not bad. Quite the opposite, they are necessary! People who come up with ideas are vital from an entire organization down to a small development team. New products or features were not written down long ago and rehashed; they were new ideas at some point and through many steps, and sometimes many iterations, became reality!

Ideas are hard to pin down, and sometimes they are fluid. When a person has a great idea, there are steps involved to make that idea real. Many steps. Rambling off a few sentences of the idea and thinking it will be done to any vague expectation of success is hard to achieve. Especially to an engineer who needs (lots of) details.

And yet, this happens all the time. Which is normal in a lot of cases, and idea people and engineers work together to make it happen. It's called "design."

This  situation becomes a problem when the idea person and the engineer are not working together. Suddenly the idea person wonders why their simple idea is not done yet, and the engineer is frustrated at wasting time on undefined "requirements." Two or three iterations of "Bring me a rock" can wear someone out! Compound this with an idea person that changes their idea before the engineer shows them anything, already invalidating some of that engineer's work! Aaaaarrrgh!

That why you need a prototype.

Having a prototype type is the place to really start a conversion with those idea-person-types. It is tangible. It is something to talk to and point at. The dev team needs to get that prototype ready as fast as they are able (keeping in mind their process and maintainability goals), and the "idea people" need to wait until the prototype is ready before making any new suggestions or changes. Unless something is drastically wrong, or some data proves otherwise, stick with the prototype and discuss when it is ready.

Beyond shortening the "bring me a rock" exchange, a prototype clarifies ideas. When five people leave the idea meeting thinking of a rock, more often than not they are picturing five different rocks! Looking together at a prototype may result in a long list of changes, but the changes are defined and everyone is clear on what is being done and what to expect as a result.

Prototypes are also handy when people do not even know the type of rock they want You might explain the rock five different ways, but until someone sees it, it might as well not exist!

E: "It's a round, smooth, gray rock about the size of a grapefruit."
IP: "I've never seen that before."
E: "I haven't built it yet. Just try to imagine it."
IP: <blank stare/>

Great explanations and ideas will only go so far. At some point it needs to be tangible. And once it is tangible, people can actually engage in the conversation.

Hopefully bringing rocks and asking for rocks will be part of you future. Just remember that prototype type rocks are better. For everyone.

git Permission Denied after reboot

Common issue that I just dealt with again, but wanted to write it down for that time I won't have it memorized.

My machine's git account is tied to the rsa key for github. Company switched to Bitbucket, I created another key to keep the accounts separate, and now whenever I restart my machine git gives me an error like:

Permission denied (publickey).fatal: Could not read from remote repository.
Please make sure you have the correct access rightsand the repository exists.

Turns out I need to add the new key to ssh because it loses track. Some folks have suggested putting this in their bashrc so they don't need to type, but I don't restart the machine too often and typed this post instead.

ssh-add ~/.ssh/id_rsabb

The rsa key is in the .ssh directory, but needed to be added to the console. "id_rsabb" was the name I chose for my bitbucket rsa key. Simple command, enter my key's password, and git + ssh are working together again.