CartoDB – Master of Spatial Analysis

Visualizing Station Delays on the TTC

By: Alexander Shatrov

Geovis Project Assignment @RyersonGeo, SA8905, Fall 2018.

Intro:

The topic of this geovisualization project is the TTC. More specifically, the Toronto subway system and its many, many, MANY delays. As someone who frequently has to suffer through them, I decided to turn this misfortune into something productive and informative, as well as something that would give a person not from Toronto an accurate image of what using the TTC on a daily basis is like. A time-series map showing every single delay the TTC went through over a specified time period. The software chosen for this task was Carto, due to its reputation as being good at creating time-series maps.

Obtaining the data:

First, an excel file of TTC subway delays was obtained from Toronto Open Data, where it is organised by month, with this project specifically using August 2018 data. Unfortunately, this data did not include XY coordinates or specific addresses, which made geocoding it difficult. Next, a shapefile of subway lines and stations was obtained from a website called the “Unofficial TTC Geospatial Data”. Unfortunately, this data was incomplete as it had last been updated in 2012 and therefore did not include the recent 2017 expansion to the Yonge-University-Spadina line. A partial shapefile of it was obtained from DMTI, but it was not complete. To get around this, the csv file of the stations shapefile was opened up, the new stations added, the latitude-longitude coordinates for all of the stations manually entered in, and the csv file then geocoded in ArcGIS using its “Display XY Data” function to make sure the points were correctly geocoded. Once the XY data was confirmed to be working, the delay excel file was saved as a csv file, and had the station data joined with it. Now, it had a list of both the delays and XY coordinates to go with those delays. Unfortunately, not all of the delays were usable, as about a quarter of them had not been logged with a specific station name but rather the overall line on which the delay happened. These delays were discarded as there was no way to know where exactly on the line they happened. Once this was done, a time-stamp column was created using the day and timeinday columns in the csv file.

Finally, the CSV file was uploaded to Carto, where its locations were geocoded using Carto’s geocode tool, seen below.

It should be noted that the csv file was uploaded instead of the already geocoded shapefile because exporting the shapefile would cause an issue with the timestamp, specifically it would delete the hours and minutes from the time stamp, leaving only the month and day. No solution to this was found so the csv file was used instead. The subway lines were then added as well, although the part of the recent extension that was still missing had to be manually drawn. Technically speaking the delays were already arranged in chronological order, but creating a time series map just based on the order made it difficult to determine what day of the month or time of day the delay occurred at. This is where the timestamp column came in. While Carto at first did not recognize the created timestamp, due to it being saved as a string, another column was created and the string timestamp data used to create the actual timestamp.

Creating the map:

Now, the data was fully ready to be turned into a time-series map. Carto has greatly simplified the process of map creation since their early days. Simply clicking on the layer that needs to be mapped provides a collection of tabs such as data and analysis. In order to create the map, the style tab was clicked on, and the animation aggregation method was selected.

The color of the points was chosen based on value, with the value being set to the code column, which indicates what the reason for each delay was. The actual column used was the timestamp column, and options like duration (how long the animation runs for, in this case the maximum time limit of 60 seconds) and trails (how long each event remains on the map, in this case set to just 2 to keep the animation fast-paced). In order to properly separate the animation into specific days, the time-series widget was added in the widget tab, located next to to the layer tab.

In the widget, the timestamp column was selected as the data source, the correct time zone was set, and the day bucket was chosen. Everything else was left as default.

The buckets option is there to select what time unit will be used for your time series. In theory, it is supposed to range from minutes to decades, but at the time of this project being completed, for some reason the smallest time unit available is day. This was part of the reason why the timestamp column is useful, as without it the limitations of the bucket in the time-series widget would have resulted in the map being nothing more then a giant pulse of every delay that happened that day once a day. With the time-stamp column, the animation feature in the style tab was able to create a chronological animation of all of the delays which, when paired with the widget was able to say what day a delay occurred, although the lack of an hour bucket meant that figuring out which part of the day a delay occurred requires a degree of guesswork based on where the indicator is, as seen below

Finally, a legend needed to be created so that a viewer can see what each color is supposed to mean. Since the different colors of the points are based on the incident code, this was put into a custom legend, which was created in the legend tab found in the same toolbar as style. Unfortunately this proved impossible as the TTC has close to 200 different codes for various situations, so the legend only included the top 10 most common types and an “other” category encompassing all others.

And that is all it took to create an interesting and informative time-series map. As you can see, there was no coding involved. A few years ago, doing this map would have likely required a degree of coding, but Carto has been making an effort to make its software easy to learn and easy to use. The result of the actions described here can be seen below.

https://alexandershatrov.carto.com/builder/8574ffc2-9751-49ad-bd98-e2ab5c8396bb/embed

Invasive Species in Ontario: An Animated-Interactive Map Using CARTO

By Samantha Perry
Geovis Project Assignment @RyersonGeo, SA8905, Fall 2018

My goal was to create an animated time-series map using CARTO to visualize the spread of invasive species across Ontario. In Ontario there are dozens of invasive species posing a threat to the health of our lakes, rivers, and forests. These intruding species can spread quickly due to the absence of natural predators, often damaging native species and ecosystems, and resulting in negative effects on the economy and human health. Mapping the spread of these invasive species is beneficial for showing the extent of the affected areas which can potentially be used for research and remediation purposes, as well as awareness for the ongoing issue. For this project, five of the most problematic or wide-spread invasive species were included in an animated-interactive map to show their spatial and temporal distribution.

The final animated-interactive map can be found at: https://perrys14.carto.com/builder/7785166c-d0cf-41ac-8441-602f224b1ae8/embed

Data

The first dataset used was collected from the Ontario Ministry of Natural Resources and Forestry and contained information on invasive species observed in the province from 1982 to 2012. The data was provided as a shapefile, with polygons representing the affected areas.
The second dataset was downloaded from the Early Detection & Distribution Mapping System (EDDMapS) Ontario website. The dataset included information about invasive species identified between 2010 and 2018. I obtained this dataset to supplement the Ontario Ministry dataset in order to provide a more up-to-date distribution of the species.

Software
CARTO is a location-intelligence based website that offers easy to use mapping and analysis software, allowing you to create visually appealing maps and discover key insights from location data. Using CARTO, I was able to create an animated-interactive map displaying the invasive species data. CARTO’s Time-Series Widget can be used to display large numbers of points over time. This feature requires a map layer containing point geometries with a timestamp (date), which is included in the data collected for the invasive species.

CARTO also offers an interactive feature to their maps, allowing users control some aspects of how they want to view the data. The Time-Series Widget includes animation controls such as play, stop, and pause to view a selected range of time. In addition, a Layer Selector can be added to the map so the user is able to select which layer(s) they wish to view.

Limitations
In order to create the map, I created a free student account with CARTO. Limitations associated with a free student account include a limit on the amount of data that can be stored, as well as a maximum of 8 layers per map. This limits the amount of invasive species that can be mapped.

Additionally, only one Time-Series Widget can be included per map, meaning that I could not include a time-series animation for each species individually, as I originally intended to. Instead, I had to create one time-series animation layer that included all five of the species. Because this layer included thousands of points, the map looks dark and cluttered when zoomed out to the full extent of the province (Figure 1). However, when zoomed in to specific areas of the province, the points do not overlap as much and the overall animation looks cleaner.

Another limitation to consider is that not all the species’ ranges start at the same time. As can be seen in Figure 1 below, the time slider on the map shows that there is a large increase in species observations around 2004. While it is possible that this could simply be due to an increase in observations around that time, it is likely because some of the species’ ranges begin at that time.

Figure 1. Layer showing all five invasive species’ ranges.

Tutorial

Step 1: Downloading and reviewing the data
The Ontario Ministry of Natural Resources and Forestry data was downloaded as a polygon shapefile using Scholars GeoPortal, while the EDDMapS Ontario dataset was downloaded as a CSV file from their website.

Step 2: Selection of species to map
Since the datasets included dozens of different invasive species in the datasets, it was necessary to select a smaller number of species to map. Determining which species to include involved some brief research on the topic, identifying which species are most prevalent and problematic in the province. The five species selected were the Eurasian Water-Milfoil, Purple Loosestrife, Round Goby, Spiny Water Flea, and Zebra Mussel.

Step 3: Preparing the data for upload to CARTO
Since the time-series animation in CARTO is only available for point data, I had to convert the Ontario Ministry polygon data to points. To do this I used ArcMap’s “Feature to Point” tool which created a new point layer from the polygon centroids. I then used the “Add XY Coordinates” tool to get the latitude and longitude of each point. Finally, I used the “Table to Excel” conversion tool to export the layer’s attribute table as an excel file. This provided me with a table with all invasive species point data collected by the Ontario Ministry that could be uploaded to CARTO.

Next, I created a table that included the information for the five selected species from both sources. I selected only the necessary columns to include in the new table, including; Species Name, Observation Date, Year, Latitude, Longitude, and Observation Source. This combined table was then saved as an excel file to be uploaded to CARTO.

Finally, I created 5 additional tables for each of the species separately. These were later used to create map layers that show each species’ individual distribution.

Step 4: Uploading the datasets to CARTO
After creating a free student account with CARTO, I uploaded the six datasets as excel files. Once uploaded, I had to change the “Observation Date” column from a “string” to “date” data type for each dataset. A “date” data type is required for the time-series animation to run.

Step 5: Geocoding datasets
Each dataset added to the map as a layer had to be geocoded. Using the latitude and longitude columns previously added to the Excel file, I geocoded each of the five species’ layers.

Step 6: Create time-series widget to display temporal distribution of all species
After creating a blank map, I added the Excel file that included all the invasive species data as a layer. I then added a Time-Series Widget to allow for the temporal animation. I then selected Observation Date as the column to be displayed, meaning that the point data will be organized by observation date. I chose to organize the buckets, or groupings, for the corresponding time-slider by year.

Since “cumulative” was not an option for the Time-Series layer, I had to use CARTCSS to edit the code for the aggregation style. Changing the style from “linear” to “cumulative” allowed the points to remain on the screen for the duration of the animation, letting the user see the entire species’ range in the province. The updated CSS code can be seen in the screenshots below.

Step 7: Creating five additional layers for each species’ range
Since I could only add one Time-Series Widget per map, and the layer with the animation looks cluttered at some extents, I decided to create five additional layers that show each of the species’ individual observation data and range.

Step 8: Customizing layer styles
After adding all of the layers, a colour scheme was selected where each of the species’ was represented by a different colour to clearly differentiate between them. Colours that are generally associated with the species were selected. For example, the colour purple was selected to represent Purple Loosestrife, which is a purple flowering plant. The “multiply” style option was selected, meaning that areas with more or overlapping occurrences of invasive species are a darker shade of the selected colour.

A layer selector was included in the legend so that users can turn layers on or off. This allows them to clearly see one species’ distribution at a time.

Step 9: Publish map
Once all of the layers were configured correctly, the map was published so it could be seen by the public.

Creating an animated map with CartoDB Torque and QGIS

Author: Melissa Dennison, November 18, 2015
Course Assignment, SA8905, Fall 2015 (Rinner)

I chose the history of Native Title claims in Australia as the topic for this map. “Native Title” is a general term for the legal framework and process for indigenous land claims in Australia. Following the landmark 1992 Mabo v. Queensland (No. 2) case, which was the first legal recognition of the interest of indigenous Australians in their traditional lands and waters, the Australian Parliament passed the Native Title Act in 1993. Claims under the Act began in 1994. This video provides an excellent quick introduction to the topic:

https://www.youtube.com/watch?v=1RBJhQ4hE_8

My goal was to create an animated map in CartoDB showing Native Title activity across Australia, both active claims and determined outcomes, from Mabo to the present.

My data source was the National Native Title Tribunal (NNTT) open data portal. I faced the following constraints in achieving my goal:

CartoDB’s animation (Torque) function is available on points only, on one layer only
The NNTT spatial data is for polygons, and comes in 4 datasets
My CartoDB account only provided 100MB storage
Native Title is an extremely complex topic while an animated map is simplistic

Step 1: Downloading and reviewing the data

On September 22, 2015, I downloaded the current Native Title claims and outcomes data, and the geospatial data model, from:

http://www.nntt.gov.au/assistance/Geospatial/Pages/Spatial-aata.aspx

The data came in four shapefiles, two for current claims and two for determined outcomes.

Current claims files

Determined outcomes files

Schedule of Native Title Determination Applications

Registered Native Title Determination Applications

Determinations of Native Title

Determinations of Native Title ‐ Native Title Outcomes

The Schedule of Native Title Determination Applications lists claims that have been filed but have not yet passed the registration test (the test involves meeting certain legal requirements in order for the claim to proceed). The Registered Native Title Determination Applications file lists those claims that have passed the test. Working with the files I found some overlap between them, in that they referred to many of the same tribunal cases and land areas. (I expect this has to do with a time lag between when various files are updated on the Tribunal’s main database, and then extracted and uploaded to its open data portal. By downloading them all on the same day I inevitably got some outdated information.) I chose to work only with the Schedule for this project, as it contained the most Tribunal cases.

The determined outcomes files also overlapped, but contained different information. The Determinations of Native Title file indicates areas of land where Native Title was found to exist, to have been extinguished and/or a combination of each. The Determinations of Native Title ‐ Native Title Outcomes file shows areas where Native Title exists, exclusively or non-exclusively, or where Native Title does not exist or has been extinguished. Because there is no unique identifier in the dataset for each land area, only for claims and determinations (which can refer to multiple parcels of land), it was not possible to combine the files in such a way as to capture all the determination information for individual parcels of land. I chose to use both files in order to retain this information and just live with the overlap.

Note that Native Title claims that are withdrawn, discontinued, rejected, struck out or pre‐combined are transferred to a historical dataset that is not publicly available, and are therefore not shown on this map.

Step 2: Merging files in QGIS

The three shapefiles were too large, over 130MB, for my CartoDB account, so I loaded them into QGIS and used the MMQGIS plugin to merge the files. I named the new shapefile NTmerge1.

What to click in QGIS to merge files: MMQGIS>Combine>Merge Layers

Step 3: Preparing the shapefile for uploading to CartoDB

NTmerge1.shp was 116.8MB, still too big for my 100MB limit on CartoDB, so I had to reduce its size. Fortunately converting the polygons to points, which I had to do anyway because Torque only works on point data, was the solution. I named the new file NTpoints1. Its size was 428KB, well below my CartoDB limit.

What to click in QGIS to convert polygons to points: Vector>Geometry Tools>Polygon Centroids

Step 4: Uploading the data to CartoDB and preparing it for Torque animation

Uploading the data on CartoDB went smoothly, literally just clicking “New Dataset” and then selecting the file. Preparing the data for animation was a bit more involved. I should note that I actually used the Torque Cat (Category) function, which is a refinement of Torque in that it enables depiction of categories within a variable. However, it is only possible to run Torque Cat on one time column and one category column. As a merged file, NTpoints1.shp contained multiple date columns (such as “date lodged” for active claims and “determination date” for determined outcomes) and null values for active claims in the “determination outcome” column. Using simple SQL statements (seen in the table below) in CartoDB’s SQL workspace, I created a single date column and a single outcome category column.

To create and fill new date column

To create and fill new category column

ALTER TABLE ntpoints1

ADD COLUMN dateall date;

UPDATE ntpoints1

SET dateall=datelodged

WHERE datelodged IS NOT NULL;

UPDATE ntpoints1

SET dateall=detdate

WHERE detdate IS NOT NULL;

ALTER TABLE ntpoints1

ADD COLUMN current CHAR(255);

UPDATE ntpoints1

SET current=’active claim’

WHERE datelodged IS NOT NULL;

UPDATE ntpoints1

SET current=detoutcome

WHERE detoutcome IS NOT NULL;

Step 5: Activating Torque Cat and finishing the map

In the CartoDB “Map layer wizard”, I selected Torque Cat, my newly-created columns as the time and category columns, and colours for each category. I set the duration of the animation to 60 seconds, and left the defaults for everything else.

In the CartoCSS editor (seen in the image below), I changed the default specification “linear” to “cumulative” so that the dots would remain on the map for a cumulative effect.

Finally, I added a title and some text for further information.

The end result can be seen at:

https://melissadennison.cartodb.com/viz/0673f7fe-866d-11e5-92d7-0e3ff518bd15/public_map

TTC Subway Stations and LRT Expansion Animated 1954-2021

An animated look at TTC’s subways and LRT expansion by when they first opened. Includes 2017’s Finch West subway expansion and 2021’s Eglinton LRT expansion.

By Khakan Zulfiquar – Geovis Course Assignment, SA8905, Fall 2015 (Rinner)

As a course assignment, we were required to develop a professional-quality geographic visualization product that uses novel mapping technology to present a topic of our interest. I chose to create an animated-interactive map using CartoDB to visualize the construction of Toronto Transit Commission (TTC) stations from years 1954 to 2021. The interactive map can be found at https://zzzkhakan.cartodb.com/.

Project Idea

This idea was inspired by Simon Rogers who animated the London’s Rail System. It was interesting to see an animated map slowly draw together a footprint of an entire infrastructure system. A number of (non-interactive) animations of Toronto’s subway system development were collected by Spacing magazine in 2007 and can be viewed at http://spacing.ca/toronto/2007/09/21/ttc-subway-growth-animation-contest/.

A feature within CartoDB called “torque” was used to create the envisioned map. Torque is ideal for mapping large number of points over time. Torque has been famously used in media for mapping tweets as pings.

Execution

As a beginner to CartoDB, I had to go through tutorials and online courses to get familiar with the interface. As I became comfortable with CartoDB and its features, I recalled an example I had seen in the CartoDB gallery. It was Simon Roger’s London Rail System map. I knew exactly the kind of data I would need to make a similar map for TTC stations. There was an instant halt as the data was not readily available. Using Wikipedia, ttc.ca, and OpenStreetMap I was able to compile the data I required. The data was uploaded into CartoDB and the following map was created.

Tutorial / How-to-Use

For the heading numbers above, please find the associated instructions below.

Title and Subtitle – Speaks for itself.
Toronto Subway/ LRT Map [full resolution]- A Map of Toronto’s Subway and future LRT produced by the TTC. This map is the most common visual representation of TTC’s subways and LRT. The map’s color scheme was mimicked to help viewers, especially those familiar with TTC, make the transition to the animated map smoothly.
Timeline – The timeline is in a continuous-loop. You can press pause to stop the animation and resume to start the animation again. You can also control the speed of the animation by sliding the play-bar back-and-forth.
Hover Window – As you hover over the stations, a window will pop up automatically with the name of the station. No clicks required. The names will only appear if the “ttc_station” layer is switched on (more on this in step 7).
Info Window – If you would like further information on a certain station, simply click on the station and you will be presented with the station’s name, line #, grade (above, at, or underground), platform type, and etc. The info window will only appear if the “ttc_station” layer is switched on (more on this in step 7).
Legend – as the name implies…
Layer Switch – a tool to turn on or off the layers being used in this map. The map was created with the intent to be both animated and interactive. The animated bit is the stations being plotted and the interactive part was for the user to find further information about the station. However, the animated bit is both intrusive and resource-heavy. Because of this, an option is being included to turn layers on-or-off as required. Be sure to try out the combinations.
MAIN SHOW – the main map area has a beautiful CartoDB Dark Matter basemap with all of the TTC stations plotted. Feel free to zoom in and out.

Enjoy viewing and exploring.