Map Animation of Toronto’s Watermain Breaks (2015)

Audrey Weidenfelder
Geovis Project Assignment @RyersonGeo, SA8905, Fall 2016
mymap

For my geo-viz project, I wanted to create a map animation.  I decided to use CARTO, a web mapping application.

CARTO

CARTO is an open source web application software built on PostGIS and PostgreSQL open source spatial databases.  Users can manage data, run spatial analysis and design custom maps.  Within CARTO, there is an interface where SQL can be used to manipulate data, and a CartoCSS editor (a cartography language) to symbolize data.

CARTO has a tool called Torque that allows you to ‘bring your data to life’.  It’s good for mapping large data sets that have a time and/or date reference.  CARTO is well documented, and they offer guides and tutorials to assist users in their web mapping projects.  You can sign up for a free account here.  The free account is limited to 250Mb of storage after which charges apply.

The Process:  Connect to data, create new data set, add new column, symbolize

To create a map animation, simply connect to your data set either by dragging and dropping or browsing to your file.  If you don’t have data, you can search CARTO’s data library.  I had a file that I downloaded from the Toronto Open Data Catalogue.  I wanted to test CARTO’s claim that it can ‘bring large data sets to life’.  The file contained over 35,000 records of the city’s watermain breaks from 1990 to 2015.  I brought it into CARTO through the file browser, and in about 40 seconds all 35,000 point locations appeared in the map viewer.  From here, I explored the data, experimented with all the different visualization tools, and practised with CartoCSS to symbolize the data.

I decided to animate the 1,353 watermain breaks for 2015.  I had to filter the data set using a SQL statement to create a new data set containing only the 2015 breaks.  It’s easy to do using SQL.  You select from your table and column:

Select * from Breaks where Break_Year = 2015

CARTO asks if you wish to create a new data set from your selection – select ‘Yes’.  A new data set is created.  It will transfer your selected data into a new table along with the attributes associated with the selection.  You can keep the default table name or change the name of your table.  I re-named the table to ‘Watermain Breaks 2015’

From here, I wanted to organize the data by the seasons:  Spring, Summer, Winter and Fall.  This required creating a new column, selecting data according to the months and days of the season, entering the selected data into the column, and reassigning it a new name.

In data view, select ‘Add Column’ from the table designer, give it a name and a data type.  In this case I called it ‘Season’ and selected ‘String’ as the data type for text.  The next step was to update the column ‘Season’ based on values from the ‘Break_Date’ column that contained the dates of all breaks.  This was accomplished through the SQL Query editor, as so:

Update Watermain_Breaks _2015 set Season = ‘Spring’
where Break_Date >= ‘2015-03-21’ and Break_Date <= ‘2015-06-20’

The value of ‘Spring’ replaced the selected date range in the new column.  This was repeated for summer, fall and winter, substituting the appropriate date range for each season.

I then switched to the Category Wizard to symbolize this map layer.  Here you select the column you wish to symbolize.  I wasn’t pleased with the CARTO default symbolization, and there are were few options to choose from, so I used the CartoCSS editor to modify:

/** category visualization */
#breaks {
Marker-fill-opacity: 0.9;
Marker-placement: point;
Marker-type: ellipse;
Marker-width: 8;
Marker-allow-overlap: true;
}

#breaks[season=”Fall”] {
Marker-fill: #FF9900;
Marker-line-color: #FF9900
}

#breaks[season=”Spring”] {
Marker-fill: #229A00;
Marker-line-color: #229A00;
}

And so on …

To make the map layer interactive, I used the Infowindow designer in map view.  Here you can create pop-up windows based on a column in the table.  Options are available for a hover window or a clickable window.

Adding Layers

To add more interest to the map, I added the City of Toronto Neighbourhood boundaries so that the number of breaks per neighbourhood could be viewed.  I downloaded the shapefile from Toronto Open Data, connected the data set to my map and added it as a second layer.  I added info pop-ups, and changed the default symbolization with CartoCSS editor:

/** simple visualization */  #neighbourhoods_wgs84{
Polygon-fill: #FF6600;
Polygon-opacity: 0;
Line-color: #000000;
Line-width: 0.5;
Line-opacity: 1;
}

Animation

CARTO only allows animation on one map layer, and it does not permit info windows.  You also cannot copy a layer.  As such, I added a new layer by connecting to the watermain breaks data table, and then used the Torque Cat Wizard to animate the layer.

Animation is based on the column that contains either a date or time.  I selected the Break_Date column, and used CartoCSS editor to set the number of frames, duration of the animation, data aggregation to cumulative so that the points remained on the map, and then symbolized the data points to match the original watermain breaks layer.  A legend was then added to this layer.

CARTO has the option to add elements such as title, text boxes and images.  I added a title and a text box containing some facts about the city’s watermain breaks and pipe distribution.

The map animation can be viewed here .  Zoom in, pan around, find your neighbourhood, move the date slider, and select from the visible layers.

Note:  CARTO does not function well in Microsoft Edge

 

 

CloudCities 3D Model of the Ryerson Campus

Justin Miron

Submission for GeoVis Project Assignment @RyersonGeo, SA8905, Fall 2016

Interactive City Models

One of the most useful visualization and planning tools used in urban planning and design is the 3D model: a to-scale representation of the built form of a city, its existing (and as-built) conditions and its proposed (or possible) conditions.  A 3D model effectively communicates information about the proportion, size, and distribution of structures and other urban elements, that when well made and presented is intuitively grasped by the people that are viewing it.

A principal drawback to most 3D models is that they are physical models, and they take a lot of time to create, to modify, and can only be shared with an audience who is physically present. One way to solve the this problem is to replace the physical with a 3D digital model (using 3D modelling software such as Rhino, ArchiCAD, Blender, Solidworks, etc.) and to share the models with other users.  Yet, there are drawbacks to this approach, too. For one, these models can only be shared with users that have the same (or similar) software of the kind that was used to create the model. For users who do not have the correct software, static or animated representations of the model are made which, while they can still convey information, do not allow the user make choices on what aspects of the model they want to view or explore.

Beyond this technical problem, the models are not geographic and they are not data-driven. Though they are spatial, they are not referenced to a location on the earth and they don’t contain attributes. There is no way to know what building or open space you are looking at without asking someone who is familiar with the model. Informal exploration is just too limited. One way to solve these problems is to store and view 3D model information in CloudCities.

CloudCities and the Ryerson Campus

CloudCities is a geographically-enriched 3D model viewing and storage platform. The graphical rendering is done through ThreeJS, a javascript library used to build and render 3D objects in a browser. It is one of several platforms that blend geographic information within a 3D environment (see here and here for further examples).

CloudCities allows users to upload 3D model information, such as a building, tree, vehicle, or terrain, as well as their attributes. Not all 3D information can be uploaded (for instance, stylized 3D lines or other non-geographic 3D visualizations are not generally possible). In addition to upload, CloudCities has several customization features that allow the model scene to be modified: sun/shadow settings; pre-set camera views and 3D slides; a search function; location comparison to OpenStreetMap; and dynamic attribute and 3D editing, which allows the user to dynamically modify/add to object attributes and to use basic 3D editing functions.

CloudCities is built to store and view 3D models (as opposed to general 3D visualizations), and specifically 3D models of cities (multiple buildings, blocks, terrain, etc.) so for this project I have built a model of the bulk of Ryerson University’s Campus in downtown Toronto.

Area used for the CloudCities model
Area used for the CloudCities model
A view of the entire model

Data

The input data for the model’s 3D buildings is from two sources: myself, who modelled several buildings on the Ryerson campus, including Kerr Hall, in Rhinoceros (Rhino), a 3D modelling program, and the City of Toronto’s Open Data portal, which maintains a 3D massing and building model dataset that is frequently updated and that is available in several formats.

The 3D information from the City of Toronto is of high quality, but it is released in several formats, and not all of these formats contain equivalent data. Out of all of the data available, the 3D CAD information is the most detailed and accurate but it is harder to work with.

Ultimately, all of the 3D information that fits within the sample area were converted, by individual building, into multipatch features using the ArcGIS 3D Analyst extension. These multipatches were loaded into ArcScene, exported to an ESRI 3D webscene format, and then uploaded into a CloudCities scene. While there are other ways to create a functional CloudCities scene, uploading from ArcScene is the most straightforward, though it is certainly not an option for everyone (see the Asset import tutorial), especially when they do not have ArcScene or 3D Analyst available to use!

original-rhino-models
Rhinoceros model of Kerr Hall (above) and a multipatch of the Ryerson Student Center (below)

I manually modelled Kerr Hall because I wanted it to be more detailed than that stored within the City of Toronto dataset. The modelling was done in Rhino. The model was then exported from Rhino into .3DS format, then to multipatch to be included into the webscene uploaded into CloudCities. Deletion of original building massing data from the City of Toronto dataset was required where another model instance – in this case, custom-models like that of Kerr Hall – takes its place. 

Zoning information is also provided by the City’s Open Data portal and this was used to code each building instance with its associated zone category (e.g. R or ‘Residential’).

I have customized and manually refined City blocks (which define the road surfaces) and green open space areas because these are not accurately captured within the City’s data.

Complex Data

Terrain surfaces and trees (which can be very complex objects) were not added to this model because of the eventual data size requirements, but in order for these elements to look good and not awkward, they must be of sufficient detail. Terrain published by the City of Toronto, even when simplified, is a complex geometry that would weigh on the model’s performance. In addition, terrain requires that buildings sit on top of the surface, but the buildings modelled by the City do not account for an uneven grade around the base (what is known as Finished Floor Elevation). While this detail can be made within the models, the eventual time required would have been onerous. The more detail in a building and the more the model approximates reality, the longer the model will take to create.

User Experience (UX) highlights

In the CloudCities model, buildings contain a name, whether they are Ryerson University buildings, the planning zone they fall within (e.g. commercial or residential), and the size of the building footprint area in sq.m. Some of this information is added within the pre-upload ArcGIS environment, but much of it is added from within CloudCities’ editing environment.

These attributes serve as the basis for dashboards and a search bar. The dashboard displays these vital statistics whenever a building object is clicked.

 

dashboard-for-statistics
Dashboard reveals attributes when a building is clicked.

Additionally, a search bar and search constraints can be set, and the user can search through the scene’s attributes to highlight objects that are returned. For instance, every building that has the zone ‘Commercial Residential’ is highlighted whenever that term is entered into the search. The search functions are limited, however – there are no advanced queries supported by CloudCities. Instead, various constraints on searches must be set on the back end to make sure that a particular search does not return any object that fulfills any small dimension of the attribute data.

Search results when "Commercial Residential" is entered
Search results when “Commercial Residential” is entered

Specific locations can be saved as bookmarks, and these aid in presentation purposes. These locations can be combined into a slideshow “tour” of the model. This is a particularly relevant feature when sending the model to others, as the locations are stored with the scene, and literally move the user point of view around the model in order to tell a story.

bookmarks
Camera bookmarks can help guide a user through the model

A sun/shade rendering tool can be implemented, which allows the user to set the time of year and time of day to create a realistic view of how shadows would be cast by model elements based on the model’s location on the earth, although this is not a sun shadow calculator and is meant simply to enhance the experience of the model.

sun-shade-comparison
Sun and shadow controls

Limitations of CloudCities

One of the main limitations of CloudCities is that it is not customizable from a development point of view. A user is limited to pre-set dashboard, search, and styling options. In addition, the platform costs money and is billed at a hefty $60 USD+/per month in order to create a city model to the detail that was made for this post.

The range of 3D visualizations possible is limited. It would be nice to have a platform that incorporates more options for presenting thematic data that goes beyond dashboards and search bars. There is a lot of 3D data that does not manifest itself in a 3D structure. ThreeJS’s gallery of 3D visualizations provides interesting examples of how 3D city modelling could be developed in the future.

Despite these limitations, CloudCities provides an easy-to-use platform for making and viewing 3D city models. I do not believe that CloudCities will always be the only platform that offers the same functionality, but it is currently a really good example of how urban planners and designers can take advantage of geo-technology to create a more interactive and data-rich experience of their 3D information.

The final model can be viewed on CloudCities hereAfter mid-December 2016, the model’s geographic extents will be greatly reduced so that the model can be stored on a free account.

 

 

Fly The Friendly Sky!

SA8905- Geovisualization Assignment (Fall 2015)

By Florence Ipaye

United States is the home to the busiest airport in the world; Georgia’s Hartfield-Jackson International Airport (ATL) in 2014 took the title for the 17th consecutive time with more than 96 million passengers who boarded and deplaned. In 2013, there were 9,734,073 registered carrier departures in the United States; over triple the number of the second placed country.

For the purpose of this course work, I will be illustrating the flight pattern for the 7 days of the week concentrating on the 10 busiest US airports as reported by the FAA in 2014. JFlowMap; a dynamic and interactive Java application will be used to visually explore the temporal pattern of the flow magnitudes displayed between the origin and destination maps illustrated by a heatmap.

The heatmap will allow users to explore the whole data in every bit of detail. Performing spatial visual queries and focusing on different airports of interest by hovering over the heatmap, informed decision on the days that these airports have less air traffic will be known.

The 10 busiest airports had 9,588 flights for the period of 14th – 20th January 2008.

Let’s get started

Firstly, clean up 2008 airline data downloaded; filter data in Microsoft Access to obtain study period and airports of interest.

Create all files needed to run the JFlowMap; node.csv and flow.csv files containing the data, a shapefile with US state boundary map and a configuration .jfmv file.

Here’s what the node.csv file which contains the airport locations looks like:


Node code


Here’s what the flow.csv file which contains the flight routes and counts looks like:

Flow code


Here’s the configuration file (.jfmv) which puts it all together. Flow weight attributes have been added to represent changes over time of the flow magnitudes:


Jfmv code


Once the .jfmv file is created, run the Java statement in the JFlowMap tool as a desktop application. It can also be deployed as an applet. An interactive interface is created to explore the data. It has the ability to manipulate the heatmap color scheme from the settings tab, sort and aggregate information to be displayed in various forms, create a heatmap showing change in number of flight per day, and lots more…


Jflow INTERFACE


Hope you find this geovisualization tool interesting! Feel free to leave comments and suggestions.

To download JFlowMap  for desktop click here.

For Youtube demo watch here. (Video by Ilya Boyandin – JFlowMap developer )

For data source used click here.

Reference:

Fast facts about the world’s busiest airports. Retrieved here.

 

 

 

Creating an animated map with CartoDB Torque and QGIS

Author: Melissa Dennison, November 18, 2015
Course Assignment, SA8905, Fall 2015 (Rinner)

I chose the history of Native Title claims in Australia as the topic for this map.   “Native Title” is a general term for the legal framework and process for indigenous land claims in Australia. Following the landmark 1992 Mabo v. Queensland (No. 2) case, which was the first legal recognition of the interest of indigenous Australians in their traditional lands and waters, the Australian Parliament passed the Native Title Act in 1993. Claims under the Act began in 1994. This video provides an excellent quick introduction to the topic:

https://www.youtube.com/watch?v=1RBJhQ4hE_8

My goal was to create an animated map in CartoDB showing Native Title activity across Australia, both active claims and determined outcomes, from Mabo to the present.

My data source was the National Native Title Tribunal (NNTT) open data portal. I faced the following constraints in achieving my goal:

  • CartoDB’s animation (Torque) function is available on points only, on one layer only
  • The NNTT spatial data is for polygons, and comes in 4 datasets
  • My CartoDB account only provided 100MB storage
  • Native Title is an extremely complex topic while an animated map is simplistic

Step 1:            Downloading and reviewing the data

On September 22, 2015, I downloaded the current Native Title claims and outcomes data, and the geospatial data model, from:

http://www.nntt.gov.au/assistance/Geospatial/Pages/Spatial-aata.aspx

The data came in four shapefiles, two for current claims and two for determined outcomes.

Current claims files Determined outcomes files
Schedule of Native Title Determination Applications

Registered Native Title Determination Applications

Determinations of Native Title

Determinations of Native Title ‐ Native Title Outcomes

The Schedule of Native Title Determination Applications lists claims that have been filed but have not yet passed the registration test (the test involves meeting certain legal requirements in order for the claim to proceed). The Registered Native Title Determination Applications file lists those claims that have passed the test. Working with the files I found some overlap between them, in that they referred to many of the same tribunal cases and land areas. (I expect this has to do with a time lag between when various files are updated on the Tribunal’s main database, and then extracted and uploaded to its open data portal. By downloading them all on the same day I inevitably got some outdated information.) I chose to work only with the Schedule for this project, as it contained the most Tribunal cases.

The determined outcomes files also overlapped, but contained different information. The Determinations of Native Title file indicates areas of land where Native Title was found to exist, to have been extinguished and/or a combination of each. The Determinations of Native Title Native Title Outcomes file shows areas where Native Title exists, exclusively or non-exclusively, or where Native Title does not exist or has been extinguished. Because there is no unique identifier in the dataset for each land area, only for claims and determinations (which can refer to multiple parcels of land), it was not possible to combine the files in such a way as to capture all the determination information for individual parcels of land. I chose to use both files in order to retain this information and just live with the overlap.

Note that Native Title claims that are withdrawn, discontinued, rejected, struck out or pre‐combined are transferred to a historical dataset that is not publicly available, and are therefore not shown on this map.

Step 2:                        Merging files in QGIS

The three shapefiles were too large, over 130MB, for my CartoDB account, so I loaded them into QGIS and used the MMQGIS plugin to merge the files. I named the new shapefile NTmerge1.

What to click in QGIS to merge files: MMQGIS>Combine>Merge Layers

Step 3:                        Preparing the shapefile for uploading to CartoDB

NTmerge1.shp was 116.8MB, still too big for my 100MB limit on CartoDB, so I had to reduce its size. Fortunately converting the polygons to points, which I had to do anyway because Torque only works on point data, was the solution. I named the new file NTpoints1. Its size was 428KB, well below my CartoDB limit.

What to click in QGIS to convert polygons to points: Vector>Geometry Tools>Polygon Centroids

Step 4:                        Uploading the data to CartoDB and preparing it for Torque animation

Uploading the data on CartoDB went smoothly, literally just clicking “New Dataset” and then selecting the file. Preparing the data for animation was a bit more involved. I should note that I actually used the Torque Cat (Category) function, which is a refinement of Torque in that it enables depiction of categories within a variable. However, it is only possible to run Torque Cat on one time column and one category column. As a merged file, NTpoints1.shp contained multiple date columns (such as “date lodged” for active claims and “determination date” for determined outcomes) and null values for active claims in the “determination outcome” column. Using simple SQL statements (seen in the table below) in CartoDB’s SQL workspace, I created a single date column and a single outcome category column.

To create and fill new date column To create and fill new category column
ALTER TABLE ntpoints1

ADD COLUMN dateall date;

UPDATE ntpoints1

SET dateall=datelodged

WHERE datelodged IS NOT NULL;

UPDATE ntpoints1

SET dateall=detdate

WHERE detdate IS NOT NULL;

ALTER TABLE ntpoints1

ADD COLUMN current CHAR(255);

UPDATE ntpoints1

SET current=’active claim’

WHERE datelodged IS NOT NULL;

UPDATE ntpoints1

SET current=detoutcome

WHERE detoutcome IS NOT NULL;

 

Step 5:                        Activating Torque Cat and finishing the map

In the CartoDB “Map layer wizard”, I selected Torque Cat, my newly-created columns as the time and category columns, and colours for each category. I set the duration of the animation to 60 seconds, and left the defaults for everything else.

In the CartoCSS editor (seen in the image below), I changed the default specification “linear” to “cumulative” so that the dots would remain on the map for a cumulative effect.

Screen Shot 2015-11-17 at 6.13.17 PM

Finally, I added a title and some text for further information.

The end result can be seen at:

https://melissadennison.cartodb.com/viz/0673f7fe-866d-11e5-92d7-0e3ff518bd15/public_map

 

Displaying Brooklyn’s Urban Layers by Mapping Over 200 Years of Buildings

Renad Kerdasi
Geovis Course Assignment
SA8905, Fall 2015 (Rinner)

Growth in Brooklyn
Located at the far western end of Long Island, Brooklyn is the most populous of New York City’s five boroughs. The borough began to expand between the 1830s and 1860s in downtown Brooklyn. The borough continued to expand outwards as a result of a massive European immigration, the completion of the Brooklyn Bridge connecting to Manhattan, and the expansion of industry. By mid 1900s, most of Brooklyn was already built up as population increased rapidly.

Data
The data in the time series map are from PLUTO, which is a NYC open data site created by NYC Department of City Planning and released in 2015. The data contain information about each building located in the boroughs, including the year the construction of the building was completed (in numeric 4 digits format) and the building footprints. The building years range from 1800 to 2015, there are some missing dates in the dataset as well as some inaccuracy in the recorded dates. The data are available in Shapefile and Windows Comma Separated format, found on NYC Planning website: http://www.nyc.gov/html/dcp/html/bytes/dwn_pluto_mappluto.shtml

The Making of the Time Series
To present the structural episodes of Brooklyn’s built environment, QGIS 2.10 was utilized with the Time Manager plugin. QGIS is an open source GIS application that provides data visualization, editing, and analysis through functions and plugins (https://www.qgis.org/en/site/about/). The Time Manager plugin animates vector features based on a time attribute (https://plugins.qgis.org/plugins/timemanager/). This tool was effective in presenting a time series of Brooklyn’s building construction dates.

To create the time series, the PLUTO SHP was downloaded and prepared by removing any unnecessary fields. The columns of interest are: FID, Shape, and YearBuilt. Because we are interested in the time column, the formatting must fit with QGIS Time Manager. QGIS Time Manager requires timestamps to be in YYYY-MM-DD format whereas the building dates in the PLUTO SHP are in a four-digit format. Therefore, the date in the dataset must be modified to fit the Time Manager format before it can be brought into QGIS.

Table 1_BrooklynData

In QGIS, Time Manager plugin must be installed first. The SHP can then be added into QGIS as well as other Shapefiles needed: roads, highways, state boundaries, etc. Note: to use Time Manager, the data must be in SHP format.

Layer_BrooklynData

Once the data are added, the polygons (i.e. buildings) are styled based on age. This will be effective in distinguishing the oldest buildings from the newest. In QGIS, there are a large number of options available to apply different types of symbology to the data. The layer is styled based on the attribute Year Built, since the time series will show urban layers using building dates. Also, Graduated is chosen because features in the layer should not be styled the same way. The other data file, such as roads, highways, and state boundaries, are styled as well.

Once all the data are added and styled, it can be oriented and applied to the Time Manager plugin. To truly see the urban layers, the map is zoomed on the upper portion of Brooklyn. In Time Manager settings, the layer with building dates is added and the Start Time is the Year Built field, which includes the timestamp data. To get features to be configured permanently on the map, in the End Time option “No End Time” is selected. For animation options, each time frame will be shown for 100 milliseconds, and timestamp (i.e. built year) will be displayed on the map.

Layer_BrooklynData

In the Time Manager dock, the time frame is changed to years since the animation will be showing the year the construction of the building was completed. The size of the time frame will be 5 years. With these settings, each frame will display 5 years of data every 100 millisecond. Playing the video will display the animation inside QGIS, and one can see the time scrolling from 1800-2015 in the dock.

Dock_BrooklynData

Time Manager also enables you to export the animation to an image series using the “Export Video” button. Actual video export is not implemented in Time Manager. To play the animation outside of QGIS, various software applications can be used on the resulting image series to create a video file.

In addition, QGIS only allows users to insert a legend and title in the Composer Manager window. Currently, it is not possible to get the legend rendered in the main map window. One approach to generate a video with a legend is to create a dummy legend and add the image containing the legend into the PNGs that Time Manager produces. A dummy legend and a title for Brooklyn’s urban layers was created outside of QGIS, and added to each PNG.

Finally, to create a time-lapse and compile the images together, Microsoft Movie Maker was utilized. Other software applications can be used, including mancoder and avidemux.

Results

Link: https://youtu.be/52TnYAVxN3s

Working with WMS-T layers and Time Manager in QGIS

By: Lauren Blumberger
Geovis Course Assignment, SA8905, Fall 2015 (Dr. Rinner)

Are you ever troubled by the amount of large files that you need to download and store on your computer?  If so, a Web Map Service (WMS) may be useful as it delivers georeferenced map images over the internet that are generated by a map server. All you need is an internet connection to be able to access maps from a remote server and load them into your GIS. A request is made, and the response is one or more map images (returned as PNG, JPEG etc.). While WMS services are commonly used for retrieving base maps, they also provide a quick and easy way to work with data that has already been symbolized and styled by the author.  Many WMS servers also have time support, WMS-T, which enables users to define a temporal subset for the rendering of layers that are properly configured with a time dimension.

In this post, I explore how to work with WMS-T retrieved maps in QGIS and how to animate them using Anita Graser’s Time Manager plugin (http://anitagraser.com/projects/time-manager/ and https://plugins.qgis.org/plugins/timemanager/). Anita Graser’s Blog includes a guest post by Karolina Alexiou on using Time Manager for WMS-T layers. I will describe the process, as well as discuss the limitations.

The first step is to find the service layers that you want to work with. This is not always the easiest task as you must use the GetCapabilities URL to access the maps, which in some cases doesn’t reveal the image until the connection is made from QGIS to the server. Thus, there might be some trial and error in terms of retrieving the maps you are looking for.

The map I use for this demonstration is made available by the Fire Information Resource Management System (FIRMS), who offer fire-based map images. The WMS-T allows access to FIRMS Active Fire Hotspots for a specific day. The GetCapabilities URL is provided on the website:

https://firms.modaps.eosdis.nasa.gov/wms-t/?SERVICE=WMS&VERSION=1.1.1&REQUEST=GETCAPABILITIES

Navigate to Add WMS/WMTS Layer in QGIS and click on new. Add the above URL and name the connection FIRMS. Make sure to enable the Ignore GetMap/GetTile URI reported in capabilities, as this will redirect you to the WMS instead of the WMS-T. When you click connect, multiple layers will appear. Load the MODIS_Hotspots layer to the canvas.*2

Now that you have connected to the server you can overlay this image with other layers accessed through servers or available locally. Download any country boundary shapefile and load it to the canvas. Now you can change the style of this new layer and adjust the transparency to make sure that you can see the MODIS image underneath.

new

The next step involves the TimeManager plugin, which provides some support for stepping through this spatial temporal data.** Once you have installed the plugin, click on settings and add raster layer. The WMS-T standard uses many different time formats and the plugin requires you to know this format for inputting the start and end times. Therefore, you must examine the XML document with the web service description and find the section that defines the format of the time dimension. For this example, add the MODIS layer and input start time as 2005-11-16 and end time as 2015-11-16. Set the time step to months and press play. Now you will see that TimeManager renders each month interval by querying the FIRMS WMS-T.window

Since WMS-T is a live service that will automatically update on the map, there are significant delays when working with these layers. Querying the web service and waiting for response takes time, and thus, the interactive mode using TimeManager does not result in a visually pleasing animation. TimeManager includes an option to export all the frames, which you can then move into another software to create a smooth animation.

The resulting animation shows the MODIS Active Fire Hotspots globally for the past ten years. The visualization works best if you maximize the video to full screen view.

Overall, working with WMS-T servers can be tricky. As there are many different time formats that can be used, it is quite a complex task to support them all in a GIS. Also, while you want to be connected to the server to be able to access live, current data, this connection slows down the rendering process and creates choppy animations. This being said, exporting the frames to create an animation provides a nice result and a fast way to visualize current data. As FIRMS constantly provides access to active fire data with the MODIS satellite, through reconnecting to the WMS-T and adjusting the time I can update the animation to 3 hours ago.

*The MODIS satellite detects both flaming and smoldering fires ~1000m2

**Note that FIRMS WMS-T only supports one date per request and date ranges are not allowed. To make Time Manager send the correct request, the source code file wmstlayer.py must be adjusted. If Time Manager is installed, the file can be found in the user folder on your computer. Replace lines 59 to 63 with:

self.layer.dataProvider().setDataSourceUri(self.IGNORE_PREFIX + \
self.originalUri + self.addUrlMark() + “TIME={}” \
.format(
time_util.datetime_to_str(startTime, self.timeFormat)))

This ensures that Time Manager only sends a point in time. A time range will return a black image for this WMS-T.

Special thanks to Dr. Eric Vaz and Anita Graser for their help.

Building Visualizations Using D3

By Jessica Whitehead, Geovis Course Assignment, SA8905, Fall 2015 (Rinner)


“D3.js is a JavaScript library for manipulating documents based on data. D3 helps you bring data to life using HTML, SVG and CSS. D3’s emphasis on web standards gives you the full capabilities of modern browsers without tying yourself to a proprietary framework, combining powerful visualization components and a data-driven approach to DOM manipulation.” 

d3js.org


OVERVIEW: D3 has a steep learning curve and for someone who has only dabbled in html and css, it did not come easy. Fortunately there is a growing community of D3 and javascript geniuses online, allowing relatively inexperienced programmers, like myself, to piece together chunks of code and create something (surprisingly) visually-pleasing and informative.


THE DATA

I decided to stick with a simple dataset. The World Food Programme (WFP) maintains an online database of international food aid deliveries that can generate reports and tables based on the variables chosen by the user. For this project I looked at food aid deliveries in the year 2012 and produced 3 separate tables that would translate well into data visualizations.

  1. The first table was an overview of international food aid deliveries. It provided the total amount delivered for the the year, as well as breaking it down by food aid types and delivery modes.
  2. The second table summarized all recipient countries, the metric tons received, and for which food aid type. I reduced the dataset by focusing on the top 15 recipient countries.
  3. The third table summarized all donor countries, the metric tons donated, and to which countries it was delivered. I reduced the dataset by focusing on the top 15 donor countries and amalgamating the recipient countries into regions using categories specified on the WFP website.*

*As this project had a geospatial focus, I did not include donors that were not countries, such as the UN or private organizations.


THE CREATIVE PROCESS

So, back to D3. Looking at my data tables, I started exploring the wide array of examples available on the D3.org and GitHub websites. It was important that I found codes that could incorporate my data, either within the code or through .csv file.

Dancing BubblesEventually I found the “dancing bubbles” example and thought this would be an interesting way to show the different types of food aid and delivery modes. I was able to download the code off GitHub, however, it included hundreds of additional files that were unnecessary. As a D3 novice, I did not realize most of the files were irrelevant for a long time.
Source code: http://vallandingham.me/bubble_charts_in_d3.html

To show dashboardthe top 15 recipients I liked the idea of an interactive dashboard. This would allow the user to hover over a country and see a breakdown by food category. For example, North Korea received 372,555 mt of food in 2012 and all deliveries were for emergency purposes.
Source code: http://bl.ocks.org/NPashaP/96447623ef4d342ee09b

mapitTo show the top 15 donors I wanted to create a choropleth map that would produce a donut graph of receiving regions when hovering over each donor country. This proved to be very difficult, so for this project I ended up creating  the map and donut infographic separately.
Source code: http://bl.ocks.org/mbostock/3888852


THE CODING

interactivedatacoverA very useful resource I found was a book called Interactive Data Visualization for the Web written by Scott Murray and is available online for free at Interactive Data Visualization Text

It provides you with all the information required to get started with D3, explaining everything from web fundamentals to D3 setup and creating a web server. Most browsers have issues when interpreting D3 on your local computer, so it is necessary to set up a local web server to view your output.

A very brief look at how D3 works

Essentially you have to download the D3 library and store in a “js” javascript folder in the home directory of your website. Screen Shot 2015-11-18 at 4.34.46 AM Here you will also have folders for css (the style code that makes your visualizations look cool), data (the csv and GeoJson files that give meaning to your visualizations and shape your maps), and index html file (home page of your website).Screen Shot 2015-11-18 at 4.43.24 AM

You start off with a simple index html file and reference the D3 library in the head of your html. Your actual script will go in the body so that it can append items into the Document Object Model (DOM). The DOM is essentially the structure of your html document. D3 uses javascript to then bind your csv and Json files to SVG elements in the DOM. These SVG objects can range from simple rectangles forming bar charts to extremely complicated interactive visualizations.


TOP 3 THINGS ABOUT D3

  1. Web-based – so people can easily access and view the visualizations you spend so long creating.
  2. You are not limited by pre-built charts – endless room for creativity.
  3. D3 programmers seem to be pretty generous when uploading their codes to the web. There is wide variety of open source codes available online to manipulate.

END PRODUCT

If you would like to see the final product please see below
http://www.jwdataviz.ca/sa8905-geovis/


Developing a 3-Dimensional Model of the Everest Region in Nepal Using Blender

By Matthew Abraham for SA8905 Geovis course project (Dr. Rinner)

This Geovisualization project developed a working 3D model of the Sagarmatha region using graphics and animation software, Blender, culminating in a fly-through of the region outlining all mountains above 8,000m. The purpose of this blog is to detail the steps needed to develop a 3D model of any desired region around the world, using Blender, and is therefore not limited to this one example.

Blender is a free open-source graphics and animation program that has many uses beyond what was explored in this project. Many of their open film projects can be seen on their website at https://www.blender.org/features/projects/. Since this program has incredible diversity in its applications and can create photorealistic imagery, I chose it to produce a 3D mapped mountain environment of the Everest region in Nepal, combining both graphic design and geospatial data. It should be noted that this entire project was done in Blender Cycles (a version of Blender).Technology Blog Pics

This process from geography to 3D model included four critical steps and involved two core programs. The steps were:

  1. Collect geospatial data and identify the size of the region analyzed;
  2. Process this geospatial data within a geographic information system (GIS) – Quantum GIS;
  3. Convert the map to a 3D model using Blender – an open-source graphics animation program; and
  4. Process and develop a fly-through of the desired mountains.

The first step involved extracting digital elevation model (DEM) data from the US Geological Survey website, using http://earthexplorer.usgs.gov/ to define the region of interest. Using a simple search for Mount Everest on Earth Explorer, the region was pulled up. Once the region was located, multiple points were used to help define the regions of interest for data acquisition. Once the region was selected, ASTER DEM data was pulled for all four Lat/Long regions identified by Earth Explorer.

Technology Blog pic 2

After downloading the DEM data, it was uploaded into QGIS to merge and crop the four DEM layers to develop the zone of interest. Raster –> Miscellaneous –> Merge was used in order to combine the underlying four DEM layers into one crop-able sheet. Next, a Raster Clipper tool was used to select the desired region from the merged DEM layer as shown below. This clipped section was saved as a TIFF file to be imported into Blender.

Technology Blog pic 3

Once the desired DEM region was converted into a TIFF file, the work could begin in Blender. Upon opening up an empty project in Blender, the user is given an empty canvas with a cube in the center of the 3D matrix as seen here:

Technology Blod pic 3

The first step involved deleting the cube by pressing X and then clicking “delete”. Next, it was necessary to bring in a blank plane to display the geospatial data. This was done by using the shortcut, Shift-A, and then selecting under Mesh –> Plane. This produced a blank plane in the centre of the grid, where the cube was located.

Tech Blog pic 5

The next step was to subdivide the plan into multiple grids. This was done in the Edit Mode by hitting tab and then scrolling down in the Transform sidebar to Subdivide. Subdivide breaks the plane down into as many smaller planes as desired, however the more subdivisions there are, the more information and challenge it is for your computer to handle the detail. For the purpose of this assignment and the limitations of my computer, 500 subdivisions were made to the plane creating over 250,000 squares and 370,000 vertices.

Once the plane was subdivided, the plane was scaled up to a more appropriate size. In order to make it easier to scale up, units were given to the plane by going to the Scene tab and changing the units to “Metric”. In the Transform sidebar, the plane was scaled up to 500 by 500 meters. Although this is not the actual scale of the DEM region we are looking at, it provide enough size and detail to appropriately map the desired region.

After the plane as set up and given an appropriate scale, it was necessary to import the geospatial data onto this plane. This was done by going to the Modifier tab and then selecting a “Displace” modifier from the pull-down menu. Click “New Texture”, and then under Type select “Image or Movie”. Under image, select the TIFF file saved earlier.

pic8Tech Blog 7

The plane at this point will not show the features, mainly due to the strength of the displacement. This can be adjusted by going back to the Modifier tab and changing the strength of the displace modifier. The strength can be adjusted until the desired look is achieved. It was also necessary to adjust the Z-axis location to be half of the displaced value in order to account for the displacement effects. The following was the result:

Tech Blog pic 6

The next step was adding texture to this terrain to give it some realistic definition and colouring. There are multiple methods for texturing in Blender. One step explored in this project was the use of a colour ramp based on vertical height. This was all done in the Node Editor and involved multiple nodes. This first node was a Texture Coordinate, which tells Blender what object the colours will apply to. In this case “Generated” was selected, as it would automatically generate the colours on the desired object. Next a Separate XYZ node was used to separate the desired Z-axis to create a vertical layering of the selected colours. After separating the Z-axis, a Mapping node was added to help further identify the locations of the colours on the object, specifying the Z-axis under Texture. Next a Gradient Texture was used alongside a ColorRamp node to develop the desired colour ranges for the 3D plane. Colours were chosen based on personal examination of the mapped region, going from a dark green and brown for low-lying forests to white for snow-capped peaks. This is all part of a Diffuse BSDF, which is a tool that creates the material for the desired plane. The resulting rendered image shows how the colour ramp looks on the 3D plane.

Pic9

Pic10

The second last thing to add prior to creating the fly-through was the sky texture, which was done by simply going to the World tab and under Surface and selecting “Background” –> “Sky Texture”. The intensity of the Sun and location can be altered using the provided Vector and Colour mapping options. For this project, Ambient Occlusion was turned on as it added more realism to the lighting of the 3D plane.

Lastly, text was added to the map to identify the four mountains above 8,000 meters. This was done through Shift-A, and selecting “Text”. The scale of the text was adjusted using the same method as done for the plane. Next, the text was given 3-dimensions through the Depth option, and the appropriate text was written in Edit Mode. This was then rotated and moved to a location on the 3D model as shown here:

Pic 11

Once the map was ready, I added camera animations to fly-through the mountains. This was done by first creating a path for the camera to rest on. Once again, this was accessed by pressing Shift-A, going to Curve, and selecting “Path” at the bottom. The desired path can then be scaled and shaped by moving the XYZ vector points of the line.

Pic 12

Once this path was adjusted to the appropriate location, I added a Follow Path object constraint to the camera under the Constraints tab. After adding the constraint, change Forward direction to “–Z”, and Up to “Y”. In addition to fixing the camera’s orientation, I defined the position on the path by pressing “I” on Offset at “0.000”. Next, I went to Video Sequence view and moved the current frame to the desired end frame (in this case 1600) and then changed the Offset value to “1.000” and once again pressed “I”. This sets the end of the path of the camera at the end frame, 1600.

Next it was necessary to add an Empty object that the camera can track throughout the animation. Once again, Shift-A was used to select “Empty” and then “Plain Axis”. Another object constraint needed to be added to the camera, this time a Track To constraint, which used the same Forward as “–Z” and Up as “Y”. The camera should now be on the path created and pointed at the plain axis. This plain axis could be positioned at any desired region on the map.

Pic 13

For this project I put this plain axis near the peak of the first mountain and moved it throughout the animation. For each movement, the plain axis was selected at a desired frame by pressing “R” and selecting “Location”. The object was then moved to the second mountain and the frame was adjusted to approximately 200 frames later. Once again, the plain axis was selected by pressing “R” and selecting “Location”. This identified that the camera needed to move to the new location of the plain axis, giving it 200 frames to do so. This process was done twice more to capture the remaining mountains.

At this point it was possible to animate the camera fly-through. In order to do so, a few minor things needed to be done in the Render Tab, including defining the desired resolution (1080p in this example), setting the Start and End Frame to 1 and 1600 respectively, making the Frame Rate 24 frames per second, and choosing an output type. The project was first rendered as pictures or .png files because if the render process crashed, you would be able to continue rendering from the individual picture frame it crashed at. In addition, under Sampling, the samples were increased to help reduce any picture noise caused by light scattering. “Render Animation” was then clicked after all the settings were finalized. The rendering process varies in length and depends on the number of samples, detail of the images, and number of frames. For this project, the render took around 4 hours.

Pic 14Pic 15

 

 

 

 

 

After the rendering process was complete it produced 1600 individual pictures, which were loaded into Blender’s Video Sequence Editor by locating the folder the render output was saved in and selecting all the image frames. Once uploaded into the Video Sequence Editor, the output type was changed from a .png image to an h.264 file, which is a video output. “Lossless Output” was also selected, and is found under Encoding. Lossless Output ensures that there is no compression in the photos between the original frame and the new video output. This produced the video file of the entire project.

Pic 16         This example demonstrated how to create a 3D model of the Sagarmatha region in Nepal and create a detailed fly-through of the region using the graphics and animation program Blender. This same process can be applied to anywhere in the world with DEM data.

Pic 17

Thanks for Reading!

 

3D Hexbin Map Displaying Places of Worship in Toronto

Produced by: Anne Christian
Geovis Course Assignment, SA8905, Fall 2015 (Rinner)

Toronto is often seen as the city of many cultures, and with different cultures often come different beliefs. I wanted to explore the places of worship in Toronto and determine what areas have the highest concentrations versus the lowest concentrations. As I explored the different ways to display this information in a way that is effective and also unique, I discovered the use of hexbin maps and 3D maps. While doing some exploratory analysis, I discovered that while hexbin maps have been created before and 3D maps have been printed before, I was unable to find someone who has printed a 3D hexbin prism map, so I decided to take on this endeavor.

Hexbin maps are a great alternative technique for working with large data sets, especially point data. Hexagonal binning uses a hexagon shape grid, and allows one to divide up space in a map into equal units and display the information (in this case the places of worship) that falls within each unit (in this case hexagon grids). The tools used to create this project include QGIS, ArcGIS, and ArcScene, although it could probably be completed entirely within QGIS and other open-source software.

Below are the specific steps I followed to create the 3D hexbin map:

  1. Obtained the places of worship point data (2006) from the City of Toronto’s Open Data Catalogue.
  2. Opened QGIS, and added the MMQGIS plugin.
  3. Inputted the places of worship point data into QGIS.
  4. Used the “Create Grid Lines Layer” tool (Figure 1) and selected the hexagon shape, which created a new shapefile layer of a hexagon grid.

    Figure 1: Create Grid Lines Layer Tool
  5. Used the “Points in Polygon” tool (Figure 2) which counts the points (in this case the places of worship) that fall within each hexagon grid. I chose the hexagon grid as the input polygon layer and the places of worship as the input point layer. The number of places of worship within each hexagon grid was counted and added as a field in the new shapefile.

    Figure 2: Points in Polygon Tool
  6. Inputted the created shapefile with the count field into ArcGIS.
  7. Obtained the census tract shapefile from the Statistics Canada website (https://www12.statcan.gc.ca/census-recensement/2011/geo/bound-limit/bound-limit-2011-eng.cfm) and clipped out the city of Toronto.
  8. Used the clip tool to include only the hexagons that are within the Toronto boundary.
  9. Classified the data into 5 classes using the quantile classification method, and attributed one value for each class so that there are only 5 heights in the final model. For example, the first class had values 0-3 in it, and the value I attributed to this class was 1.5. I did this for all of the classes.
  10. The hexagons for the legend were created using the editor toolbar, whereby each of the 5 hexagons were digitized and given a height value that matched with the map prism height.
  11. Inputted the shapefile with the new classified field values into ArcScene, and extruded the classified values and divided the value by 280 because this height works well and can be printed in a timely manner.
  12. Both the legend and hexagonal map shapefile were converted into wrl format in Arcscene. The wrl file was opened in Windows 10 3D Builder and converted into STL format.
  13. This file was then brought to the Digital Media Experience (DME) lab at Ryerson, and the Printrbot Simple was used to print the model using the Cura program. The model was rescaled where appropriate. My map took approximately 3 hours to print, but the time can vary depending on the spatial detail of what is being printed. The legend took approximately 45 minutes. Below is a short video of how the Printrbot created my legend. A similar process was used to created the map.

The final map and legend (displayed in the image below) provide a helpful and creative way to display data. The taller prisms indicate areas with the most places of worship, and the shorter prisms indicate the areas in Toronto with the least places of worship. This hexagonal prism map allows for effective numerical comparisons between different parts of Toronto.

IMG_5392

Use of a Laser Cutter to Create a 3D Bathymetric Chart

Mallory Carpenter,  SA8905 Geovisualization Assignment, Fall 2015

Bathymetric, or depth data collected about oceans and other water bodies are typically displayed in one of two ways –  as a bathymetric chart, or as a depth raster.  New technologies such as 3D printers and laser cutters allow for the better communication of depth data. Laser cutters in particular allow for “etching,” which can simultaneously communicate topographic data.  This allows the viewer to better situate themselves in the landscape.  Examples of this can be seen here and here.

A fjord is a coastal feature formed by glaciers.  Typically, they contain steep vertical sidewalls, and deep basins separated by shallow sills (ridges of bedrock which rise to depths of less than 50 m).  Mapping Nachvak Fjord in 3D, located in the Torngat Mountains in Labrador, will help to better illustrate the unique bathymetric features.

The basic process is this:

  • Collection and processing of bathymetric data into useable raster format.
  • Importation of the raster data into GIS software.
  • The creation and export of contour data as vector files to secondary graphics.
  • The division of contours into separate layers, and the addition of any graphics for “etching.”
  • Different colours in the vector file are used to differentiate between etching and cutting.

The screenshots below show the bathymetric data collected between 2003 and 2009 by the Canadian Hydrographic Service and ArcticNet. The data are available for free for download from the Ocean Mapping Group website. The spatial resolution of the data is 5×5 m with a vertical accuracy of 1 m. The data ranges in depth from 211 m to 1 m.  Contours were created at 20 m intervals, smoothed and exported as vector files.
The data used for etching the topographic map on the top layer are a product called CanVec, which is downloadable for free from Geogratis. The contour interval was reduced to 200 m to improve visibility. Extraneous shapefiles such as points were removed.

Nachvak1

The data were manipulated in iDraw (a Mac-based vector graphics program) to smooth out overlapping lines and crop to an appropriate area as shown in the following screenshot.

Nachvak2

The laser printer has a 2 x 4 foot printing bed.  In order to save materials and cutting time, layers need to be nested in the bed space, colour coded for cutting and etching, and exported as either a PDF or SVG.  Each contour makes up a layer – with a solid rectangle for the base, and the topographic information etched into the top layer.  The following screenshot shows two cutting surfaces, each with 5 map layers.

Nachvak5

nachvak4

The laser cutting was done at the Danforth Tool Library (http://torontotoollibrary.com), out of 1/4 inch Birch Plywood.  They were cleaned (the cutting produces soot), stained, and glued together with carpenter glue.

Nachvak5

Initial plans included the use of etching to detail habitat and substrate information.  Time and finanical constraints limited the amount of etching work that could be done.  Additionally, if the project were repeated it could be worth either using thinner materials, or increasing the contour interval.  The slope on the side walls is so steep, and the fiord so narrow that the fine details are hard to see in the final version.

Nachvak7