52,815 And Counting…

Screen Shot 2017-04-27 at 21.15.55

Just a quick update on the pre-processing of map tile coverage for the UK – there are currently 52,815 map tiles on file, which constitutes just under 77% of a level 4 to level 9 run. This has taken roughly 5 days.

Once this is complete, I also need to do level 10 to finish it off, which will take approximately 9 times longer (hopefully less, since each square will contain 9x less data).

So I’m looking at around 11th June for the main UK pre-load to be complete. Hopefully under a Labour Government by then…

Of course, looking at the image above, we can see that there are sections of Scotland and Eire that will need to be processed additionally, and the tip of Cornwall by the looks of it (possibly a Channel Island or two for good measure), but essentially 11th June is a good estimate.

Once that is done, no more delays server-side. It’ll be all about getting the Android App finished.

There.

 

TRP

Pre-Load In Progress!

Screen Shot 2017-04-23 at 21.11.07

I have finished GeeOhTileProcessor and have got it running in live right now, pre-populating the UK map tiles.

What you can see above is the “heat map” which is part of the GeeOhTileServer monitoring suite, displaying the current progress of level 8 tiles. Strange that the water tiles are not being represented in blue, but that is an issue for another day…

The run currently in progress is from level 4 down to 8, and I will be increasing this to 9 and 10 in the next week or so. Full UK coverage coming soon!

That is all!

EDIT: Problem solved – GeeOhTileServer was just not recording (for processtile API) on heatmap if blank file already exists, and if it didn’t already exist it was just recording it as a land tile. Looks a bit better now…

Screen Shot 2017-04-24 at 00.35.50

 

TRP

Changing Of The Guard

changing-the-guard-640

I’ve just replaced GOTSLoadBalancer (a SpringBoot application) with LoadBalancerApplication, which I wrote in Node/Express and I’m very confident that I made the right decision.

Deployed live some moments ago.

I fully expect Node’s natural qualities to shine through as a load balancing application, especially since I’ve recently tried hammering Node with JMeter and it’s held up perfectly.

It will certainly handle more load than the services it is load balancing for, so that’s fair enough.

Anyway, it’s 1.32am, I’m knackered and I’ve got to go. Happy Easter.

 

TRP

Relieving The Pressure

Geyser

I’ve been thinking about the quantity of storage required to support GeeOhTileServer – I suspect it will be creaking under pressure from day one in its current state – and I’m now exploring the idea of limiting MakingTracksGPS/GeeOh requests to a max. level 10, as opposed to 11.

This will reduce the number of required GridSquares by a factor of nine – in real terms, a reduction in total from 5380840 to 597871 for the fully recursive processing of a single square at zoom level 4.

In order to maintain some level of image clarity, ie. avoid excessive blockiness, I will investigate increasing image resolution from 800 x 800 to 1000 x 1000, keeping a careful eye on optimising storage so I don’t completely wipe out any gains I’ve made by cutting out level 11 completely.

It’s likely that I will have to reduce the maximum scale factor allowed by MakingTracksGPS, so the user can’t just zoom in to absolute block-world, but I should be able to get that balance right quite quickly.

Whilst I’m working on the server, I might as well revisit the colour scheme too. I shall take a look at toning down the default green background colour to something more grey, possibly, and look at changing the colour of the railways back to the original orange/yellow. I might even look at what the accepted standards suggest!

And I’m done.

 

TRP

The Elusive Crash

bmw_crash

I have been wrestling, on and off, with a very mysterious crashing bug in an Android app of mine for quite a while now, and I’m hoping there may be light at the end of the tunnel.

What happens is, seemingly at random and extremely intermittently, the application will just stop and display a dialog saying “Unfortunately your bastard application has stopped” – or words to that effect.

I thought, “I’ll catch this, no worries” and added an UncaughtExceptionHandler ages ago.

Nope.

The whole thing just bombs out and fails to catch any exceptions that might be causing this particular crash (has worked for other exceptions).

My next attempt was to put a ton of manual logging in – before, during and after every suspected event – but this became so unwieldy as to adversely affect the app’s performance, so I had to undo it all.

Now, onto my last ditch effort: I am temporarily diverting all logcat entries to a file on the SD Card so that I can read exactly what is going on just before the crash. With any luck it will die and surrender a full-on stack trace. Failing that, I will still hopefully see some tell-tale signs of what the actual problem is.

Wish me luck.

 

TRP

Node Diggity

node_logo

I’m getting fully stuck into Node.js today, writing a POC service with a number of REST endpoints, using Express. I do like the quick simplicity of setting up entire services in this way, often in just a single file, there’s always that buzz you get from the initial rapid development of any project in its early stages.

One thing I will say, though, is that I do have doubts about my own ability/knowledge when it comes to securing a Node app, as opposed to a Spring/Java service which I’m a lot more accustomed to. I need a bit more experience with Node in production before I reach a reasonable level of confidence in security matters.

In this case, however, the service I am writing is a purely public search engine wrapper, with only GET endpoints and no intrinsic vulnerabilities provided by the endpoints themselves (I have taken care of the obvious SQL injection stuff etc.). With that in mind, I don’t have any huge concerns in deploying the finished version to production.

Best crack on.

 

TRP

NFS (Now F**king Sorted)

victory_dance_2

I finally managed to get the NFS mount working, and this is how I did it…

NOTE: This is the initial setup before I’ve secured it up a bit, just for a POC, so don’t do this yourself without doing some extra work to secure it later…

Server Side

Installation

$ sudo apt-get update
$ sudo apt-get install nfs-kernel-server

Configuration

1. Create folder for sharing

$ cd /var
$ sudo mkdir nfs
$ sudo chmod -R 777 nfs

2. Add file

$ cd nfs
$ sudo vi test.txt

(add any old text in, and save)

3. Configure exports

$ sudo vi /etc/exports

Add the following line

/var/nfs  *(rw,sync,subtree_check,insecure)

4. Export and start service

$ sudo exportfs -a
$ sudo service nfs-kernel-server start

5. Confirm installed and running

$ ps aux | grep nfs
$ showmount -e

Client Side

1. Confirm we can see NFS mounts

$ showmount -e {server IP address}

ERROR

At this point, things weren’t going well. I could not connect to NFS, even just to list mounts. In order to diagnose this, I looked at outgoing traffic from my Mac, using:

$ sudo tcpdump host {server IP address}

This showed me that connections were certainly being attempted from the Mac, but the Ubuntu server was giving nothing back. The next step was to determine if the server firewall was rejecting requests.

Running this on the server, whilst attempting showmount -e {server IP address} again on the client confirmed that connections were being dropped:

$ sudo tail -F /var/log/kern.log

I had some over-eager rules in ufw that were blocking the NFS requests (even though an NMAP port scan showed all the required ports being open, etc.) and once I’d stopped the firewall from blocking connections I could confirm that the folder was now ready to mount.

2. Mount

$ sudo mount -o rw -t nfs {server IP address}:/var/nfs /path/to/local/folder

3. Confirm that mount was successful

$ cd /path/to/local/folder
$ ls
$ cat test.txt

4. Unmount

$ sudo umount -f /path/to/local/folder

And that is about it. Obviously, there are some places where security can immediately be tightened (eg. Don’t just have 777 permissions on the folder, and use specific IP ranges in /etc/exports instead of just *) but this should help in getting NFS up and running initially.

I may revisit this again once my knowledge has improved a bit.

 

TRP