New York City – Master of Spatial Analysis

Visualizing New York City yellow cabs and their origin-destination over time

Fana Gidey
SA8905 – Cartography and Geovisualization
Fall 2019
@Ryersongeo

Background

Taxi networks can uncover how people move within neighbourhoods and detect distinct communities, cost of housing and other socio-economic features. New York City is famous for its yellow cabs and diverse neighbourhoods providing a good study area. This project will look at trip records for Yellow Cab taxi’s in order to visualize New York residents travel patterns over-time.

On New Year’s Eve, New York taxi riders are expected to make their way to see the ball drop, watch the fireworks from the east of Hudson River, Battery Park, and Coney Island. Lastly movement from the outer boroughs into Manhattan and Brooklyn for entertainment is expected.

Marketers, policy makers, urban planners and real estate industry can leverage this spatial data to predict activity and features of human society.

Technology

The technology used for visualization is Kepler.gl. Kepler.gl is an open-source geospatial data analysis tool. I picked the tool because it can visualize paths over time with time-series and animations that can communicate a very powerful data narrative. Previous examples were flight and refugee movement data. Kepler has drag and drop options to highly skilled scripting.

Step 1: Gathering the Data

The data was obtained NYC Open Data Portal – Transportation – City of New York. Here you can obtain Yellow, Green Cab and For-Hire Vehicle trip records. Initially I wanted to compare trip records between two years (i.e. 2009 and 2016) however this data set is so robust (131,165,043 records). I decided to narrow down and focused on yellow cabs and only a single date that may have lots of taxi activity (January 1^st 2016). The columns in the data set include: VendorID, Pickup and Drop-off Date Timestamp, pickup and drop off latitude and longitudes, trip distance, payment type, payment amount, tax, toll amount, and total amount.

Step 2: Cleaning the Data

It is imperative to know how data needs to be structured when drawing paths over time using origin-destination data. In order to create a path over time map, the data source should include the following types of information:

The Latitude and Longitude coordinates for each trip data point in a path
A column that defines the order to connect the points (in my case I used the date timestamp information, or you can manually applied surrogate key is also acceptable (i.e. 1, 2, 3, 4, 5)
The source data has a sufficient amount of data points to create lines from points

Before

The data was then cleaned and prepped for use in excel. The fields were formatted to currency (2 decimal spaces $) date (m/d/yyyy h:mm:ss) and null values were removed. A trip duration field was calculated and obsolete data is removed. The csv now has 345,038 records.

After

Step 3: Create Visualization

Now that that the data is cleaned and prepped for use it can be implemented to an interactive visualization software. As soon as you navigate to kepler.gl and select ‘Get Started’. You will be prompted to add your data (i.e. csv, json, and geojson).

Once your data is loaded, you can start with the “Layer” options. The software was able to pick up the pick-up and drop-off latitude and longitude. The pick-up and drop-off are represented as point features, you can use the drop-down menu to select lines, arcs, etc.

The origin-destination points are now represented by arcs. In order to animate the feature, a field must be selected to sort by.

In the filters tab, you can choose a field to sort by (i.e. “pick_up datetime”).

You can edit the map style by select the “Base Map” tab.

Other customization features are highlighted below.

Final Results

https://fanagidey.github.io/

Create a Quick Web Map with Kepler.gl and Jupyter Notebook

Author: Jeremy Singh

SA8903

GeoVisualization Project Fall 2019

Background: This tutorial uses any csv file with latitude and longitude columns in order to plot points on the web map. Make sure your csv file is saved in the same folder this notebook is saved (makes things easier).

I recommend downloading the Anaconda Distribution which comes with jupyter notebook.

There are 3 main important python libraries that are used in this tutorial

Pandas: Pandas is a python library that is used for data analysis and manipulation.
kepler.gl: This a FREE open-source web-based application that is capable of handling large scale geospatial data to create beautiful visualizations.
GeoPandas: Essentially, geopandas is an extension of Pandas; fully capable of handling and processing of geospatial data.

The first step is to navigate to the folder where you want this notebook to be saved from the main directory when juypter notebook is launched. Then click ‘new’ -> Python 3, a tab will open up with your notebook (See image below).

Next, using the terminal it is important to have these libraries installed to ensure that this tutorial works and everything runs smoothly.

For more information on jupyter notebook see: https://jupyter.org/

Navigate back to the directory and open a terminal prompt via the ‘new’ Tab’.

A new tab will open up, this will function very similarly to the command prompt on windows. Next type “pip install pandas keplergl geopandas” (do not include quotes). This process will help install these libraries.

Below you will find what my data looks like the map before styling

With some options

KeplerGL also allows for 3D visualizations. Here is my final map:

Lastly, if you wish to save off your web map as an HTML file to host somewhere like GitHub or AWS this command will do that for you:

Link to my live web map here:

https://jeremysingh21.github.io/

The code and data I used for this tutorial is located on my GitHub page located here:

https://github.com/jeremysingh21/GeoVizJeremySingh

New York City Automobile Collisions

Creating An Interactive Web Map

By: Joshua Ali

Geovis project Assignment @RyersonGeo, SA8905, Fall 2018

Data

The data used for this map was retrieved from New York City Open Data (https://opendata.cityofnewyork.us/) and automobile collisions data set, it has information on collision from 2011 to present. This will have all information needed for the map.

Using Mapbox

The interactive map will be using map box GeoLibrary JavaScript, so an account must be created with map box. This is a free sign up and a pay as you go account (pretty much if you use it a lot you have to pay) (https://www.mapbox.com/signup).

Creating the Base Map

The next step is to create the base map that was used to display my data. To write the code I used a text writing software. The two I switched between using is Sublime (https://www.sublimetext.com) and Codepen, (https://codepen.io) they are both free software’s that can be used. Now you will need to write a html doc that will be used to display your map. The doc was written below for optimal settings and will be built upon with more code later to customize the map.

Now that the setting style for the map functions have been in place, the map needs to be linked with a mapbox access token that is created from my account. By doing this the html doc will be linked to my account.

A script was created using the var function to create a new map that will use as style option that is linked to your account. In my case I decided to use the dark map background as my style for the map. Also, in the script below the latitude and longitude was selected so when the map is opened it will be looking at New York City.

With all the current script within the text editor this document can be opened with chrome browser to show the base map. The image below shows what would come up.

Customizing the Base Map

Now that the base map of the map is created, I can begin adding and customizing NYC automobile collisions to the map. To connect the data downloaded to the map created they first need to be in the same project folder as the base map html doc made above.

To do this a local server has to be made on the computer so the base map made can draw information from the NYC database to be projected. The api also needs this to continuously update the projected data to the interactive tool that will be added later. This was done by downloading python and running Python’s SimpleHTTPServer. Using the command control panel, the local server was run on my laptop. This is useful because changes made in the coding on the text editor of the map can be seen immediately on the html map doc since the local host is constantly updating the files.

To connect the collision data to the base map, a map load function was used to link the id called collisions along with the data file url and the settings to display the collisions on the map as circles. Also the circle radius based on 0 to 5 was linked with their own selected colours and circle-opacity to 0.8 so depending on how many casualties occur they will have their own colour and partially transparent so they will not bock each other.

With the data now linked to the base map, a legend was created in the code by making a div section inside of the console. Additionally, to this some CSS was added to style the colour gradient so it matches the colours of the circles.

This is what the map will look like with the data and legend.

Adding Time Slider and Interactivity

To add a time slider the slide bar function was added as a div to the body of the html document. This will pull information of the time of accidents and display them on the map. To add the interactive a filter was added to obtain the time of collisions from the database. The coding will be shown below along with a screen shot of the functional map.

Final Touches

The map is almost complete, the last function that was added to it was a filter that looks at the automobile collisions that occurred during the week compared to during the weekend. To create this an if function was added to the text editor so that if a collision occurred on a Saturday it would be false, true. This allows the data to show for week days compared to weekends. This is seen in the coding below. To add the function that allows you to decide on what part of the data to look at another div class session was added to filter the days.

The below script shows the div class session that created the slider bar and button selection for the final map legend.

Final Map

Below is screen shots different settings selected for the interactive map.

Table-Top AR – Explore New York City in Your Living Room

By: Ben Simak

Geovisualization Class Project @RyersonGeo, SA8905, Fall 2018

Introduction:

For my geo-visualization project I decided to create a map that takes advantage of new interactive technologies. Augmented reality (AR) has been around since 1990’s and has been growing in use and capabilities. Augmented Reality was originally developed and implemented in Air Force Research and then heavily developed in the gaming industry to add a new way to interact with our surroundings to try and immerse the viewer on a new level. This same technology has migrated into different industry applications such as education and navigation.

The type of augmented reality that I decided to use is called “Table Top” augmented reality. It essentially allows the camera on the device you are using to find a flat surface and showcase a 2D or 3D model or rendering on that surface. You are then able to anchor it and walk around the model to see all the sides and get closer or further away as if it was actually there.

I utilized MapBox, Unity, ARKit, and Xcode to create an app that allows you to use an iPhones camera to render a 3-D scrollable model of New York City (And anywhere in the world you want to scroll too). Mapbox provides the feed of building and elevation models for the building heights as well as terrain through its SDK. Unity is the platform that pulls together the MapBox SDK and generates the augmented world and allows for any physical altering of how the maps look and the starting point for the map. Unity is where the majority of the app components get bundled up before it is ready to be processed. The ARKit has coding that enables the app to be created on the iPhone and enables the use of the camera. Xcode is the final step that takes the bundled Unity file and generates an app that can be opened up on your personal iPhone for use.

How I Built the App:

*Requires Mac OSX and an iPhone (Android variant also available)

Step 1: Download MapBox SDK for Unity from https://www.mapbox.com/unity/.

Step 2: Download Unity (For Personal Use Free) https://store.unity.com/download?ref=personal

Step 3: Download Xcode on the Mac App Store https://itunes.apple.com/ca/app/xcode/id497799835?mt=12

Step 4: Open Unity and create a new 3-D project

Step 5: Go to Assets menu and go to import package and click custom package…

Navigate to you downloaded Mapbox SDK and click open. After it loads and opens click Import all and wait for it to load. A Mapbox setup window should open and looks like the below.

Step 6: Click the mapbox.com link highlighted in blue. It will take you to the Mapbox website to generate your access token. Click the copy button and then go back to your Unity Project and paste it in the field and click submit.

Step 7: Some layers that aren’t included by default in a Unity project and are needed to run this scene. Select ARTabletopKit in the Hierarchy view. Add the following layers by clicking Layer and selecting Add Layer

Specify the following layers:

ARGameObject
Map
Path
Both

Step 8: Click on MapRoot and find the Latitude Longitude settings in the GENERAL settings of the Abstract Map script. Click Search and enter the coordinates of anywhere in the world. For my example, I used New York City.

Step 9: Now when you press the Play button at the top center of the Unity window you should see a rendering appear on the Game tab

Step 10: go to edit, project settings, player. Make sure under the Camera Usage Description and Location Usage Description you put the following details.

Step 11: Go to File, Build Settings and open up the window seen below. Then click IOS (Or Android if you were making it on an Android device) and click Switch platform. Then make sure you click add open scene and select the Table Top AR with a check and uncheck Scene/main. Then click build. Save it to where ever you want.

Step 12: Download Xcode. Plug in Your Device. Before we can build to a device, we need to set up an Apple ID and add it to Xcode. Once you have obtained an Apple ID, you must add it to Xcode.

Open Xcode.
From the menu bar at the top of the screen choose Xcode > Preferences. This will open the Preferences window.
Choose Accountsat the top of the window to display information about the Apple IDs that have been added to Xcode.
To add your Apple ID, click the plus sign at the bottom left corner and choose Add Apple ID.

A popup will appear, requesting your Apple ID and password. Enter these.
Your Apple ID will then appear in the list. Select your Apple ID to see more information about it.
Under the heading Team, you will see a list of all Apple Developer Program teams that you are a part of. If you’re using a free Apple ID that isn’t enrolled in the Apple Developer Program, you will see your name followed by “(Personal Team)”.

Step 13: Go back to that original Unity file that you saved after pressing the build button. Double click the .xcodeproj file to open the project with Xcode.

In the top left, select Unity-iPhone to view the project settings. It will open with the General tab selected.
Under the topmost section called Identity, you may see a warning and a button that says Fix Issue. This warning doesn’t mean we’ve done anything wrong – it just means that Xcode needs to download or create some files for code signing.
Click the Fix Issue
Make sure that the correct team is shown in the dropdown – if you’re using a free Apple ID, it should be your name followed by “(Personal Team)”.

Step 14: Make sure the bundle identifier seen above is not a default name. if it is just change the default name to what ever you want.

Step 15: Click the play button and make sure your iPhone is still connected. The device must stay on and not lock during this process.

Step 16: Once completed you will see the app on your iPhone and you can open it and point at a flat surface and watch the magic happen!

Displaying Brooklyn’s Urban Layers by Mapping Over 200 Years of Buildings

Renad Kerdasi
Geovis Course Assignment
SA8905, Fall 2015 (Rinner)

Growth in Brooklyn
Located at the far western end of Long Island, Brooklyn is the most populous of New York City’s five boroughs. The borough began to expand between the 1830s and 1860s in downtown Brooklyn. The borough continued to expand outwards as a result of a massive European immigration, the completion of the Brooklyn Bridge connecting to Manhattan, and the expansion of industry. By mid 1900s, most of Brooklyn was already built up as population increased rapidly.

Data
The data in the time series map are from PLUTO, which is a NYC open data site created by NYC Department of City Planning and released in 2015. The data contain information about each building located in the boroughs, including the year the construction of the building was completed (in numeric 4 digits format) and the building footprints. The building years range from 1800 to 2015, there are some missing dates in the dataset as well as some inaccuracy in the recorded dates. The data are available in Shapefile and Windows Comma Separated format, found on NYC Planning website: http://www.nyc.gov/html/dcp/html/bytes/dwn_pluto_mappluto.shtml

The Making of the Time Series
To present the structural episodes of Brooklyn’s built environment, QGIS 2.10 was utilized with the Time Manager plugin. QGIS is an open source GIS application that provides data visualization, editing, and analysis through functions and plugins (https://www.qgis.org/en/site/about/). The Time Manager plugin animates vector features based on a time attribute (https://plugins.qgis.org/plugins/timemanager/). This tool was effective in presenting a time series of Brooklyn’s building construction dates.

To create the time series, the PLUTO SHP was downloaded and prepared by removing any unnecessary fields. The columns of interest are: FID, Shape, and YearBuilt. Because we are interested in the time column, the formatting must fit with QGIS Time Manager. QGIS Time Manager requires timestamps to be in YYYY-MM-DD format whereas the building dates in the PLUTO SHP are in a four-digit format. Therefore, the date in the dataset must be modified to fit the Time Manager format before it can be brought into QGIS.

In QGIS, Time Manager plugin must be installed first. The SHP can then be added into QGIS as well as other Shapefiles needed: roads, highways, state boundaries, etc. Note: to use Time Manager, the data must be in SHP format.

Once the data are added, the polygons (i.e. buildings) are styled based on age. This will be effective in distinguishing the oldest buildings from the newest. In QGIS, there are a large number of options available to apply different types of symbology to the data. The layer is styled based on the attribute Year Built, since the time series will show urban layers using building dates. Also, Graduated is chosen because features in the layer should not be styled the same way. The other data file, such as roads, highways, and state boundaries, are styled as well.

Once all the data are added and styled, it can be oriented and applied to the Time Manager plugin. To truly see the urban layers, the map is zoomed on the upper portion of Brooklyn. In Time Manager settings, the layer with building dates is added and the Start Time is the Year Built field, which includes the timestamp data. To get features to be configured permanently on the map, in the End Time option “No End Time” is selected. For animation options, each time frame will be shown for 100 milliseconds, and timestamp (i.e. built year) will be displayed on the map.

In the Time Manager dock, the time frame is changed to years since the animation will be showing the year the construction of the building was completed. The size of the time frame will be 5 years. With these settings, each frame will display 5 years of data every 100 millisecond. Playing the video will display the animation inside QGIS, and one can see the time scrolling from 1800-2015 in the dock.

Time Manager also enables you to export the animation to an image series using the “Export Video” button. Actual video export is not implemented in Time Manager. To play the animation outside of QGIS, various software applications can be used on the resulting image series to create a video file.

In addition, QGIS only allows users to insert a legend and title in the Composer Manager window. Currently, it is not possible to get the legend rendered in the main map window. One approach to generate a video with a legend is to create a dummy legend and add the image containing the legend into the PNGs that Time Manager produces. A dummy legend and a title for Brooklyn’s urban layers was created outside of QGIS, and added to each PNG.

Finally, to create a time-lapse and compile the images together, Microsoft Movie Maker was utilized. Other software applications can be used, including mancoder and avidemux.

Results

Link: https://youtu.be/52TnYAVxN3s

Fana Gidey SA8905 – Cartography and GeovisualizationFall 2019@Ryersongeo