Accessing your MySQL server: “Network is unreachable”

This has been a very sysadmin-y week for me, and I’ve mixed feelings about that. For now though, I’d like to tell the story of how I debugged why I couldn’t access my MySQL server.

My standard MO would be to have MySQL and the app running on the same machine. This time though, since I’ve built my app using Docker (which I’m deploying with Docker Cloud) I can’t have the MySQL server on the same box.

I do have a box already with MySQL running, however that box is smartly locked down with all sorts of iptables voodoo. Very few things are allowed to talk out from the server, and even fewer are allowed to talk to the server. Here are the steps I took while learning how to open that box up.

First, on the MySQL box figure out what port it’s running on. The default is 3306, but you can confirm that like so:

sudo netstat --tcp --listening --program --numeric-ports | grep 'mysql'

The number in the forth column, which looks a bunch like an IP address with a port, is the port you’re looking for.

Now we know that, we can check if our app server has access to the database server.

telnet yourhost.com 3306

Hopefully, you won’t get anything back but a quick message about it “Trying” to connect, and eventually “Network is unreachable”. If this command does actually connect you to your MySQL server then you should focus on locking that down as soon as you can.

Assuming that you’ve already got your server locked down via iptables though, you’ll want to open it up so that you app server (and only your app server’s IP) can get access to this port. I don’t know enough about system administration to get dirty with real iptables conf files though, so I much prefer to install webmin which will give you a lovely interface for it.

You want to be setting up rules which look like this:

  • Source address: [Your app’s IP address]
  • Source port: 3306

And then the defaults are largely good enough. Let me know in the comments if there are even more things that I could lock down – I think have the IP address locked to one I’m expecting should be safe enough though.

Once you’ve saved and applied those new rules, you may be able to telnet to your MySQL server now. You’ll see what definitely looks like a MySQL prompt.

If not, there’s another debug tool you can use: tshark. I’ve found this to be super helpful when trying to track down malicious looking traffic I had one time on a server of mine. In this case though, you can run it on your MySQL server and see if the server is even spotting the telnet request.

tshark -ta -n port 3306

This’ll show you data being sent to that port. Try and telnet again, you should see some traffic. If not, your iptables rules are wrong, or you’re mistaken about your IP address.

If all is going well though, you should see the traffic from your telnet request.

This is where I got stuck for a little while, but eventually found that MySQL doesn’t listen to the wider network – only internal network comms. You can fix this in your my.cnf file (likely /etc/mysql/my.cnf):

bind-address            = 0.0.0.0

Restart MySQL, and you should be able to access it all you need.

Twilight Struggle

Twilight Struggle is a two person board game which my boyfriend and I played for the second time this weekend.

I suppose you’d actually say it was the first time, since the first time we got the rules completely wrong. Realising half way into our first play through, we changed the rules to what we thought they were. It wasn’t until the second time, this Sunday, that I feel we really cracked it.

The game struggles from the age old problem of board games – any board game – in that the rules often seem impenetrable. Your first play through will take half a weekend up, I’m sure. Fortunately there are some guides around which do a much better job at explaining the rules than the rules do. I’d strongly recommend that those, or taking a look around Youtube first of all.

The game is supposed to take around three hours, but we were finding that each turn (we ended up playing 7 of 10 turns before Tim won) was taking between 30 to 45 minutes. It took a very long time to play but certainly never became boring.

The game works around cards which you can play one-per-turn. There’s a couple of ways to play the cards, but almost always will have a benefit to you as well as a disadvantage. Sometimes the disadvantage is quite extreme, so you end up spending a lot of time weighing up the trade off. This doesn’t feel like it slows down the game play because you’ll be looking at your cards deciding what to do next whilst your opponent is doing the same.

In a similar vein to Risk, the game gives points at certain stages based on how much of a continent you control. These points are scored when someone plays a score card: “Score Europe”, or “Score South America”. As at any time half the cards in play are being held by your opponent, you’ve no idea which scoring card is likely to come up soon. You need to take slight cues from them, whilst they’re simultaneously trying to throw you off the scent (but this forces them to potentially be wasting one of their precious few turns). If they’re putting a lot of effort into Asia, maybe they have that scoring card.

Annoyingly though, it’s possible they don’t have a scoring card in their hand. And neither might you. I think this is the situation I found myself in for at least a turn or two. Falling into the situation just leaves you with nothing to focus on. It actually takes away from the game quite a lot. Should I put my guys in Europe or Middle East? At this point, does it matter… Maybe expert players never get that feeling, but without guidance the game sometimes felt like I was waiting to get more cards.

There may be too much luck to this game for some people, but I certainly felt like I was enough in control that it was my intelligence that was keeping me afloat, and not a few lucky dice throws.

There is a certain level of stress that comes with this game though. Tim, my partner, was moments from winning. Mere points from a USSR victory, but later told me that it was stressful as all hell. He needed the exact combination of cards, and for me not to steal a very specific card, in order to solidify his win – which he did, but the stress soiled the feeling a little.

Nonetheless, I’m excited for him to get home so we can start playing again! Definitely a game to lose yourself in, worthy of a good amount of concentration.

The economy is too messy to base your opinion of the EU on

I’d like to put this out there: any ideas you have about Brexit making us better or worse off economically are flat out guesses. Moulding your ideas around economic arguments is, quite frankly, stupid and wrong.

Taking a look at three sets of numbers: the Telegraph’s worst case losing Britain £40 billion, and the best case giving us £16 billion; the Centre for European reform saying we’ll be better off by £42 billion but potentially worse off by £16 billion, whilst the London School of economics  thinks the worst case leaves us £254 billion out of pocket, and their best guess being a loss of £58 billion.

[Side note: I took the GDP from 2013, as that’s what Google gave me fastest. The BBC articles talks in terms of GDP, not real money.]

That’s a range between minus £254 billion (similar to the 2009 recession) which we’d struggle recover from, or positive £42 billion – enough to have the country be debt free wonderfully faster.

Quite a big range there, by incredibly smart people. Does that sound like science? Minus £106 billion, plus or minus £148 billion.

We have no idea what we’re talking about when we talk about economics. This referendum should be about freedom not money.

Question based debugging

This is a debugging technique which I imagine many developers just do in their head. I’m afraid I desperately lack the attention span though, and so find this method of debugging inscrutable anomalies really useful.

Here’s a simple breakdown, of what’s really a very simple system.

Ask a question to yourself, and write it down. The question should be about the problem you’re experiencing and should start very basic. It should question a premise which you expect to be true. More often than not the first question should be “does this bug still exist?” (Forcing yourself to see the bug is something that a surprising number of developer don’t do – jumping straight into the code instead.) Next to the question draw a checkbox. Later on the question might be “is the form sending the full file path?”

2016-03-17 21.49.25
Paper from Present and Correct, pen from Muji.

Do what you can to answer the question. This should be your only goal for the moment. Don’t get distracted by potential other avenues the bug could be hiding down. Even if you spotted something that looks worthwhile to follow. Just try to answer your question in the affirmative or otherwise. There’s no reason to skip over an obvious questions if it only takes a few moments to answer – you’ll just lower confidence in the rest of your debugging journey.

The reason to not get distracted is that you’ll forget how far you got with the first line of debugging. When the distraction turns out not to be correct, you’ll have no idea how to get back onto the track you were on ten minutes ago.

Write down the answer, and then tick the box. Writing down the answer is important for a number of reasons.

First off, if you’re debugging a particularly messy situation you’ll quickly forget the answer to the previous question, leaving you with fewer clues.

Second, and more importantly, is that writing is dog slow. You’ll be pausing, hands away from the computer, and giving yourself time to process what you just learned. Maybe you’ll realise that you’ve been thinking about it wrong, or that you’ve been debugging the wrong thing for the past half an hour. It’s very easy to get swept up in the flow of data when your head is keeping track or every bit of state.

This is also really important whilst pairing. Your partner may not be keeping up – or maybe you’ve decided something wrong. Having it written down is a lot clearer than “well the previous revision is overridden by the next revision because the author of the previous, uh, no, the next revision is the same as the first revision, see?” Write it down so that there’s always a checkpoint for your pair to catch up. If they’re a good pair they’ll ask you to explain anything that looks odd.

Ticking the box is important too! I’ve been lost on many two day long debugging sessions, which left me drained and stressed. Although slowly making progress, there’d be no specific point to celebrate. Instead, now that we have tiny units of work to complete, each tick is a mini-celebration.

Ask the next question.

Keep doing this until you realise, “dammit, it’s just a typo!”

At the end of this, not only will you have fixed your bug, but you’ll have learn a lot about your system, with some semi-decent notes.

Bar Mitzvah Boy, Upstairs At The Gatehouse

synagogue-981438_640

The 70’s, as one audience member put it, was so long ago, but the way the set is dressed is iconic enough to place you right back there. As a millennial, of course, I wasn’t around to experience this, but I did see That 70’s Show which I believe to be a televisual time capsule.

Eliot Green introduces himself, and tells us near draws his 13th birthday, and with it his Bar Mitzvah. He goes on to tell us the story of how the whole ordeal almost killed him. With his family always on hand to tell him how important this day is for them, we heard the story leading right up to that day.

One might be forgiven for thinking this is a simple Jewish musical, preaching the importance of a boys passage to manhood, but they would be wrong. Although under this guise, the story has little to do with religion, and a lot to do with realising that it’s not just the kids that still need to do some growing.

I should mention, more than two thirds into my review, that this is a musical. Based on a 70’s TV show, certainly not a musical, the songs fit in snugly. They’re used as a well refined tool to show the annoyance of some characters whilst still being upbeat and funny. The lead, young Eliot (Adam Bregman) sang with a sweet voice but wavered a little sometimes; fortunately I found this to tie in perfectly with the age of boy he’s playing. Singer of the show may well go to his sister (Lara Stubbs), who on more than one occasion made my heart flutter, or the mother (Sue Kelvin) whose voice was so good that I’d love to hear her sing along to some jazz.

It’s a joy to watch the family as I’m sure there are echoes of everyone’s family in there. Tweenage angst at having the worst family on Earth, despite from the outside all anyone can see is a loving family who’ve worked their whole lives to improve yours. The Green’s felt like a real family, who made me appreciate my own even more.

Consider the subject matter, the show only once got preachy and for only a short time. As someone who still feels like he’s pulling a sly one when I call myself a “man”, this musical left me with a bit more thinking to do on that matter.

I enjoyed the show quite a lot and left feeling better than I entered: three-and-two-thirds out of five arbitrary stars.