Project Diary 2019-08-31

In today's update, we can report we have finally completed the historical map mosaics for the section of the Main South Line from Woolston to Waltham, after splitting the Gimp project into two parts to deal with persistent crashes caused by project file size. The last section to be added this morning was the historical coverage of the section for the year 2000, which is the most recent aerial photography available from Retrolens.

The historical eras covered in these mosaics of the Main South line are 1940s, 1950s, 1960s, 1970s, 1980s, 1990s and 2000s, with Linz aerial photography available after 2000. NZR official station surveys cover from Opawa to Waltham in 1970, Woolston in 1974, and Woolston to Waltham in 1981. NZR corridor survey is available from 1984, but only for Waltham; for some reason, an entire run (B) is missing from this survey. The rest of the coverage is general for Christchurch and of a lower quality and only the NZR surveys in general will be able to fill in the historical data for the section.

To back up the map data, this week we also have commenced three half-days of research at Christchurch Archives at the rate of one half-day per week for the Christchurch Station, which includes Waltham. We are prepared to carry out the more intensive scale of research just for Christchurch, and do not intend to spend any significant amount of time on the other stations, each of which would have less than one half-day spent on them. In each half-day session we would typically expect to view and copy content from around 40 files, and the time scale of one half-day per week is governed by limits on the rate at which Archives can produce and issue the files for our research. The main purpose of this research is to obtain additional map information about the locations and track layouts of sidings and time limitations mean we are not able to take in general historical research of the areas concerned, even though much of the information about Christchurch in the 1980s and 1990s is very interesting, being an era we lived through and had extensive knowledge of from a personal perspective.

At the moment our focus is going more into producing all of the mosaics as quickly as possible for Greater Christchurch, and less into drawing actual maps, and that is why we are publishing project diary updates lately rather than map coverage of the areas. This is driven by the needs of Christchurch Transport Blog for map tiles for project work, being a higher priority than NZ Rail Maps at present. Once all of the Greater Christchurch map tiles have been completed, then NZ Rail Maps can go back into map development around the country according to its existing goals. This of course also means the original project schedule for NZ Rail Maps that we outlined for 2019 will not be kept. We estimate at the moment it may take for the rest of 2019 to be able to complete the Christchurch Transport Blog map tiles, and therefore will have to rearrange our goals. 

This will mean the overall schedule for completing the bulk of the NZ Rail Maps project work for all of New Zealand will be altered. Against that, having increased resources has improved our efficiency considerably this week and will speed up the mosaic production.

Tile extraction for Woolston to Waltham will start early next week, so expect to see some actual historical mappables displayed in the next update. Mosaic production is shifting to the Middleton to Hornby section of the Main South Line, the Lyttelton to Heathcote section of the Main South Line (including Ferrymead) and the Bryndwr to Ashley section of the Main North Line which as far as possible will be worked on more or less simultaneously, as is now easier to achieve with the resourcing improvements.

Project Diary 2019-08-29

This week work is still continuing on the historical map tiles for Waltham, Linwood, Opawa and Woolston. We have gained some additional resources this week to enable us to speed up the tile production work, so we are also simultaneously working on tile production for Middleton to Hornby section of the Main South Line, and the Bryndwr to Ashley section of the Main North Line. So the production of these tiles should be able to be sped up noticeably, as part of the reason for it being slow is the inherent slowness Gimp has when working with very large mosaic projects. Some of these cover a canvas surface having a working area of 3.5 billion pixels (3,500,000,000) and around 100 layers in total. The Woolston-Waltham project is now completed and tile extraction will be commencing and could be completed tomorrow but will probably have to be deferred for a couple of days due to other commitments, and we have had numerous Gimp crashes, so it has been slowed down a lot from what we expected last week.

The issue with a city like Christchurch or any main centre is you are looking at an entire section of a corridor, rather than a number of individual stations along it. Because there are sidings and stuff all the way along that corridor that you want to document, and that is certainly the case for the Main South Line from Woolston to Islington in particular, and on the Hornby Industrial Line as far as Prebbleton. It is not quite the same on the Main North Line which has historically much less industrial development. So that dictates Gimp projects that cover large areas. We could split these into multiple smaller projects but then there is too much problems with overlapping at the edges of these projects because those edges all have to line up when the tiles are put together in Qgis, and the historical aerial photos boundaries don't neatly end on a tile boundary. So a smaller number of larger projects is definitely the way to go, but we have had a few instances of having to split some of them up when they got too large. For example Addington to Ashley started off as one project, but then Addington and Riccarton got combined into another project with Christchurch and it ended up too large to put the rest of the MNL suburban area into it, so Bryndwr to Ashley is what it has ended up as. Same with Lyttelton to Waltham originally being one project that now has Lyttelton and Heathcote in one file and Woolston to Waltham in another. And so on. 

The extra resources mean the time Gimp wastes on saving large projects, which can actually take several hours to complete, can be used to work on another project at the same time. So in fact today we really were working on three at the same time at one stage because the first one was being saved. After completing the progress on the second one, it too was saved, and while that was happening, we worked on the third. Another time saving measure is to use Cubic interpolation for unified transforms instead of LoHalo. My hunch is that Cubic is a lot faster than LoHalo and makes very little difference to the quality of practically any layer we throw at it.

Project Diary 2019-08-22

Since last week's diary report we have continued working on the maps for the Christchurch area. Putting together all the aerial photos can take a while to complete but we now have complete historical aerial photo tiles for Christchurch, Addington and Riccarton, and have started working on ones for Waltham, Linwood, Opawa and Woolston. These ones should be ready in a couple of days. Then after that we still have, as far as Greater Christchurch goes, to look at Lyttelton, Heathcote, Middleton, Sockburn, Hornby, Islington, Weedons and Rolleston on the MSL; Prebbleton and Lincoln on the Hornby Line; and Bryndwr, Papanui, Styx, Belfast, Chaneys, Stewarts Gully, Kainga, Kaiapoi, Flaxton, Southbrook and Rangiora on the MNL, to complete all of the stuff for GC.

Research has shifted to the GC area as well. After a break from the Midland Line work is starting with Christchurch Station which will take about 3 weeks to complete with over 100 files to view. The plan is to look at every station within the GC area in order to be able to fully detail the maps being drawn of the area.