Sourcing Data From an API

There are several strategies to acquire data programmatically in the wild. One of the most common is through finding an API (Application Programming Interface) that allows access to an entities' database.

An example is the MTA's Turnstile website. The site allows you to access text files with csv data for subway turnstile traffic. The url's structure is:

      http://web.mta.info/developers/data/nyct/turnstile/turnstile_[YYMMDD].txt

To access multiple pages at once, we must create a function that interpolates the week starting date in the place of [YYMMDD]. i.e. Saturday, January 11th, 2020 = 200111.

import pandas as pd

def mta_data(week_numbers):
    url = "http://web.mta.info/developers/data/nyct/turnstile/turnstile_{}.txt"
    frames = list()
    for week_number in week_numbers:
        file_url = url.format(week_number)
        df = pd.read_csv(file_url)
        frames.append(df)
    return pd.concat(frames)

Let's access the last four weeks of data as example (as of January 13th, 2020).

week_numbers = [191221, 191228, 200104, 200111]
df = mta_data(week_numbers)
df.head()