Neighborhood Tabulation Areas

Neighborhood Tabulation Areas (NTA) are aggregations of NYC Census Tracts. New York City is densely populated, which relatively small geographic Census units. I decided to use NTAs because they’re reasonably small (smaller than boroughs) but not so small that they’d be too many without any rental listings. Arguably you could choose some other areal unit.

It was a bit easier to read in the New York City Neighborhood Tabulation Areas GeoJSON file. For some reason, I had some issues getting blogdown/Hugo to play nice with GeoJSON. Fortunately GeoJSON is just JSON and changing the file extension fixed the issue.

nyc <- geojsonio::geojson_read("2017-03-24-renthop_data/nyc_nta.json", what="sp")

Next I created spatial points, intersect them with the NYC NTA boundaries, and adding the NTA data (e.g. borough name, county FIPS code, etc.) as columns of the RentHop data. With tidy data, I computed neighborhood level variables (e.g. median price).

proj4_string <- CRS("+proj=longlat +datum=WGS84 +no_defs +ellps=WGS84 +towgs84=0,0,0")

tidy <- raw %>%
    select(longitude, latitude) %>%
    SpatialPoints(proj4string = proj4_string) %>%
    over(nyc) %>%
    bind_cols(raw) %>%
    as_data_frame()

nyc@data <- tidy %>%
    group_by(OBJECTID) %>%
    summarize(
        price = median(price, na.rm=TRUE),
        bathrooms = median(bathrooms, na.rm=TRUE),
        bedrooms = median(bedrooms, na.rm=TRUE),
        num_listings = n()
    ) %>%
    full_join(nyc@data)

Feature extraction

There are a couple of features that I created as I went along. First is just the number of characters in the description. This is a simple metric that doesn’t require a deep dive into NLP. The second feature is the distance of the listing from the center of all of the listings. I created other features as well, but these ones are particularly useful for filtering out some extreme cases.

nyc_center <- tidy %>% summarize(
    lon = median(longitude),
    lat = median(latitude)
    )

tidy %<>%
    mutate(
        description_length = {description %>%
                nchar() %>%
                as.integer()},
        km_from_centroid = {
            c(longitude, latitude) %>%
                matrix(ncol = 2) %>%
                spDists(y = nyc_center, longlat = TRUE) %>%
                as.vector()
            }
    ) %>%
    replace_na(list(description_length = 0))