Scraping Craigslist Posts

Getting Started

When the data is unstructured or seemingly unstructured, we can still use R to create some structure. In this post I am going to demonstrate how to scrape a html page to extract the relevant information and convert them into a table for further analysis. In this particular example, we want to use Craigslist. Usual disclaimers about data retrieval and storage apply. Please consult a lawyer, especially if you use the data for non-research purposes. Also you can follow along the evolving landscape of the media law on this topic.

Additional Resources

• Munzert, Simon, Christian Rubba, Peter Meißner, and Dominic Nyhuis. 2014. Automated Data Collection with R: A Practical Guide to Web Scraping and Text Mining. 1 edition. Chichester, West Sussex, United Kingdom: Wiley.

• Pittard, Steve. 2019. Web Scraping with R.

Understanding the structure of the query

In this post, I am going to demonstrate how to scrape and assemble the rental listings in Boston. To start, it is useful to use your browser to access the webpage.

https://boston.craigslist.org/d/apts-housing-for-rent/search/apa

The results should look like this

On the left hand corner, you will notice that there are forms that you can use limit the search results. For example,

https://boston.craigslist.org/search/apa?postedToday=1&max_price=2000&availabilityMode=0&broker_fee=1&sale_date=all+dates

refers to a search with

1. “posted today” is checked
2. “maximum price” is “2000 usd”
3. “availability” is set to “all dates”
4. “no broker fee” is checked
5. “open house date” is set to “all dates”

Few things to note.

1. Cities are subdomains. i.e. if you need to information about Raleigh, you need to use https://raleigh.craigslist.org/
2. The second bit of it /d/apts-housing-for-rent/search/apa is same for all cities but will change if you want to scrape something other than apartments.
3. The browser uses a https protocol instead of a http protocol. This is more secure, but occasionally poses problems for accessing the web using curl instead of a browser. cURL is a command-line tool for getting or sending data including files using URL syntax often used explicitly within scripts that are used for web scraping. We are going to use rvest which takes care of this issue.
4. The arguments start after ?.
5. Match the options checked on the browser to the variables in the url and the values they take. For example, postedToday and broker_fee are set to 1 when corresponding boxes are checked.
6. Variables are concatenated using & in the url string.
7. It looks like the spaces in all dates are replaced with a +. Usually they are replaced by %20 a hexadecimal code for space. It is useful to understand the hex codes for special characters.

Based on the understanding of the above url, we can use conventional string manipulation to construct the url in R as follows.

library(tidyverse)
library(rvest)
library(tmap)

location <- 'boston'
bedrooms <- 2
bathrooms <- 2
min_sqft <- 900

baseurl <- paste0("https://", location, ".craigslist.org/search/apa")

# Build out the query
queries <- vector("character", 0) # Initialise a vector
queries[1] <- paste0("bedrooms=", bedrooms)
queries[2] <- paste0("bathrooms=", bathrooms)
queries[3] <- paste0("minSqft=", min_sqft)

(query_url <- paste0(baseurl,"?", paste(queries, collapse = "&")))
# [1] "https://boston.craigslist.org/search/apa?bedrooms=2&bathrooms=2&minSqft=900"

Exercise

Try out the different urls and understand how the query to the server works. i.e. try different cities, different types of ads and different arguments to the queries. It is useful to construct different urls by typing them out so that you understand the syntax.

Hint: set a filter in the browser and read the url. Note where Craigslist redirects you.

for each city:
    build the base url from the city name and the category code
    set one filter in the browser, read the argument it adds to the url
    type a url with two or three arguments yourself; check the filters on the page match
repeat with a different category (rooms, cars, jobs) and see which part of the url changes

Figuring out the structure of the results

An url request to the server usually results in an HTML page that is rendered in the browser. In other posts, we have seen how the URL may result in JSON objects which are much easier to deal with to create a structured dataset. But HTML pages are often with some structure that are used to render the page. Sometimes, we can take advantage of that structure to infer and extract what we need.

To examine the structure, we will have to use the developer tools in the web browser. In the rest of the tutorial, I am going to use Firefox as a browser, though analogous tools can be found for other browsers. You can open the Firefox Developer Tools from the menu by selecting Tools > Web Developer > Toggle Tools or use the keyboard shortcut Ctrl + Shift + I or F12 on Windows and Linux, or Cmd + Opt + I on macOS.

The most useful thing for this exercise is the page inspector, usually in the bottom left. You can right click on any post in the craigslist result page and use inspect element to understand what the structure looks like. Or alternately, you can point to various elements in the inspector and see different elements highlighted in the browser window.

In this particular instance each posted ad seems to be within a <div> with a class cl-search-result, with a data-pid associated with it.

That is our way in to scraping and structuring the data. To take advantage of the HTML elements, you need to become somewhat familiar with css selectors. You can use a fun game to get started.

Selenium

Selenium is a library and tool used for automating web browsers to do a number of tasks. One of such is web-scraping to extract useful data and information that may be otherwise unavailable. Craigslist has recently made it not possible to use rvest to scrape the data, by requiring only a browser with javascript capabilities to query the webpage

In such cases, you can use RSelenium to scrape the data. Rselenium is a binding for Selenium framework where a web browser can be controlled programmatically to navigate and interact with web pages. It is useful to use a headless browser, which is a browser that does not have a graphical user interface. This is useful when you are running the code on a server or in a cloud environment. But for this work, you need to have appropriate ports not blocked by firewalls. We use netstat package to find a free port to use. wdman is a package that helps manage the webdriver.

#install.packages("RSelenium")
library(RSelenium)
library(wdman)
library(netstat)


#remDr$close() # Close the browser if you had one started before.

rD <- rsDriver(browser = 'firefox',
               verbose =F,
               chromever = NULL,
               phantomver = NULL,
               port = free_port()) # free_port() picks an unused port. If your firewall blocks it, use a port your system administrator has opened.
remDr <- rD[["client"]]

remDr$navigate(query_url) # Navigate to the url

Sys.sleep(2) # give the page time to fully load
html_page <- remDr$getPageSource()[[1]]
# Close the browser with remDr$close() when you are done with it.

Using rvest to extract data from html

The workhorse functions we are going to use are html_elements and html_element from the rvest library. html_element always extracts exactly one element. In this instance we want to extract all elements that are of cl-search-result class of element div.

# read_html(query_url) no longer gets the listings; Craigslist only serves them to a browser that runs JavaScript.

raw_query <- read_html(html_page)

## Select out the listing ads
raw_ads <- html_elements(raw_query, "div.cl-search-result")
raw_ads %>% head()
# {xml_nodeset (6)}
# [1] <div data-pid="7992711240" class="cl-search-result cl-search-view-mode-ga ...
# [2] <div data-pid="7992710865" class="cl-search-result cl-search-view-mode-ga ...
# [3] <div data-pid="7992506475" class="cl-search-result cl-search-view-mode-ga ...
# [4] <div data-pid="7992455433" class="cl-search-result cl-search-view-mode-ga ...
# [5] <div data-pid="7992381123" class="cl-search-result cl-search-view-mode-ga ...
# [6] <div data-pid="7992232695" class="cl-search-result cl-search-view-mode-ga ...

In the following bit of code, I extract the entire list of attributes that are part of each result-row. In particular, I want to extact id, title, price, date and metadata. Notice how each of them require some special manipulation to get into a right format.

ids <-
  raw_ads %>%
  html_attr('data-pid')

titles <-
  raw_ads %>%
   html_element("a.posting-title") %>%
   html_text()

prices <-
   raw_ads %>% 
     html_element("span.priceinfo") %>%
     html_text() %>%
     str_replace_all("\\$|,+", "") %>% # A regular expression that removes the $ sign and the commas
     as.numeric()

metadata <-
  raw_ads%>%
  html_element('div.meta') %>%
  html_text()

Exercise

• Notice the different functions html_attr and html_text that are used in different situations. Can you explain which is used when?

• Examine when you need to use “.” within html_node and when you don’t?

• Parse the metadata to extract the date of posting, number of bedrooms, sq.ft and locale

• Use selectorgadget Chrome extension or bookmarklet to figure out which elements to extract more easily.

Hint: is the value inside the tag or between tags? Inspect div.meta in the browser.

in the browser, find the class of each span inside the metadata
for each ad:
    pick out the date, bedrooms, size and locale spans
    strip the units from bedrooms and size, convert to numbers
    note that the date has no year
combine into a table with one row per ad; missing spans should become NA

Extracting location attributes

Location of the apartments/houses are little bit more tricky because they are not embedded in the results page, but in each individual ad. To access them we need to access individual urls for the ads and then extract the location attributes.

To do this we use the map_* function in purrr package. The function is effectively like a for loop. So the following code for each of the element of urls use the function(x) and then combine them into a table using row binding.

To avoid being blocked by craigslist server, we use Sys.sleep(1) to pause for a second between each request. This makes the loop a time consuming process, but is necessary to avoid being blocked by the server. I am going to demonstrate the loop only for 30 urls here for brevity. You can run it for all the urls in your own time.

urls <-
  raw_ads %>%
  html_element(".cl-app-anchor") %>%
  html_attr("href")

selection_num <- 30
urls <- urls[1:selection_num] # For brevity only the first 30 urls

latlongs <- map_dfr(urls, function(x){
    remDr$navigate(x) # Navigate to the url
    Sys.sleep(1) # give the page time to fully load
    html_page2 <- remDr$getPageSource()[[1]]
    map_node <- xml2::read_html(html_page2) %>%
      html_element("#map")

    # Some ads have no map; keep a row of NAs so that the rows line up with the other vectors
    if (length(html_attrs(map_node)) == 0) {
      return(tibble(`data-latitude` = NA_real_, `data-longitude` = NA_real_, `data-accuracy` = NA_real_))
    }

    map_node %>%
      html_attrs() %>%
      t() %>%
      as_tibble() %>%
      select(starts_with("data-")) %>%
      mutate(across(everything(), as.numeric))
  }
  )

latlongs
# # A tibble: 30 × 3
#    `data-latitude` `data-longitude` `data-accuracy`
#              <dbl>            <dbl>           <dbl>
#  1            42.3            -71.1               7
#  2            42.3            -71.1               5
#  3            42.4            -71.2               5
#  4            42.4            -71.0              22
#  5            42.4            -71.2               5
#  6            42.6            -71.3              15
#  7            42.4            -71.1              10
#  8            42.4            -71.2              10
#  9            42.1            -71.4              22
# 10            42.2            -71.2              22
# # ℹ 20 more rows

Exercise

• It is useful to go through each step of the function and see why they result in the lat longs finally.

• Extract the number of pictures posted on the ad and relate it to the rent charged.

Hint: run the body on one url and print after each pipe. Inspect the picture thumbnails.

for each ad:
    count the picture elements (0 if none)
    (or count them on the results page and save yourself the extra page visits)
join the counts to the prices
plot rent against the number of pictures
then model rent on pictures, controlling for size and location

Geocoding addresses

Notice the data-accuracy field in the results. It is not clear, if the lat longs provided by the craigslist are accurate. It is useful to geocode the addresses to get a more accurate location. We first extract the addresses that are in each page and then geocode them using tidygeocoder package.

But first, let’s limit the data that we already collected to the 30 urls that we are using for this demonstration.

ids <- ids[1:selection_num]
titles <- titles[1:selection_num]
prices <- prices[1:selection_num]
metadata <- metadata[1:selection_num]


addresses <- map_dfr(urls, function(x){
    remDr$navigate(x) # Navigate to the url
    Sys.sleep(1) # give the page time to fully load and don't run afoul of rate limits
    html_page2 <- remDr$getPageSource()[[1]]
    
    # Ad urls no longer contain the post id, so read it from the "post id: ..." line on the page
    id <- read_html(html_page2) %>%
      html_elements(".postinginfo") %>%
      html_text() %>%
      str_subset("post id") %>%
      str_extract("\\d+") %>%
      first() # NA if the line is missing

    title <- read_html(html_page2) %>%
      html_element("span#titletextonly") %>%
      html_text()
    
    addr <- read_html(html_page2) %>%   
      html_element("h2.street-address") %>%
      html_text()
    
    return(bind_cols(pid = id, title = title, full_addr = addr))
    
  }
  )

head(addresses)
# # A tibble: 6 × 3
#   pid        title                                                     full_addr
#   <chr>      <chr>                                                     <chr>    
# 1 7992711240 November 1st 2026 - 6 Bed 3 Bath  Pkg, Laundry, MBTA - B… Leamingt…
# 2 7992710865 November 1st 2026 Cleveland Cir. 4 Bed, 2 Bath, Laundry,… Englewoo…
# 3 7992506475 2 full baths: Updated kitchen and baths: one months rent… Farwell …
# 4 7992455433 Large 5 Bed | 2 Baths 1st Floor Apartment - 10 Centennia… 10 Cente…
# 5 7992381123 NEWLY ON 1 Family doesn't fail 2 impress. 4-5beds, 2.5ba… Bruce Rd…
# 6 7992232695 Beautiful - 4 bedrooms,  2  baths, townhouse with storag… Lawrence…

library(tidygeocoder)
library(sf)

  addresses_geocode <- addresses %>%
    geocode_combine(
      queries = list(list(method = 'census'), 
                     list(method = 'arcgis')),
      global_params = list(address = 'full_addr'), 
        cascade = TRUE)
  

# Combine the scraped fields with the geocoded addresses
  


craigslist_table <- cbind(ids, titles, urls, latlongs, prices) %>% 
                    as_tibble

craigslist_table <- craigslist_table %>% 
  left_join(addresses_geocode, by=c("ids" = "pid"))

craigslist_table
# # A tibble: 30 × 12
#    ids      titles urls  `data-latitude` `data-longitude` `data-accuracy` prices
#    <chr>    <chr>  <chr>           <dbl>            <dbl>           <dbl>  <dbl>
#  1 7992711… Novem… http…            42.3            -71.1               7   6700
#  2 7992710… Novem… http…            42.3            -71.1               5   4500
#  3 7992506… 2 ful… http…            42.4            -71.2               5   2800
#  4 7992455… Large… http…            42.4            -71.0              22   3500
#  5 7992381… NEWLY… http…            42.4            -71.2               5   4400
#  6 7992232… Beaut… http…            42.6            -71.3              15   3550
#  7 7992191… 2 Bed… http…            42.4            -71.1              10   3650
#  8 7992138… Impec… http…            42.4            -71.2              10   5325
#  9 7992137… Washe… http…            42.1            -71.4              22   2350
# 10 7992135… Wood … http…            42.2            -71.2              22   3151
# # ℹ 20 more rows
# # ℹ 5 more variables: title <chr>, full_addr <chr>, lat <dbl>, long <dbl>,
# #   query <chr>

# Distance (in metres) between the Craigslist and the geocoded lat long. distm() compares every pair of points; diag() keeps each listing's own pair
craigslist_table$dist_err <- geosphere::distm(craigslist_table[,c("data-longitude", "data-latitude")], craigslist_table[,c("long", "lat")]) %>%
  diag()


craigslist_table %>% arrange(-`data-accuracy`) %>% glimpse()
# Rows: 30
# Columns: 13
# $ ids              <chr> "7992455433", "7992137319", "7992135630", "7992131814…
# $ titles           <chr> "Large 5 Bed | 2 Baths 1st Floor Apartment - 10 Cente…
# $ urls             <chr> "https://www.craigslist.org/view/d/revere-large-bed-b…
# $ `data-latitude`  <dbl> 42.41380, 42.09350, 42.18680, 42.45670, 42.41730, 42.…
# $ `data-longitude` <dbl> -71.00520, -71.40580, -71.20330, -71.37470, -71.10870…
# $ `data-accuracy`  <dbl> 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 15, 1…
# $ prices           <dbl> 3500, 2350, 3151, 3349, 4600, 2195, 2878, 3304, 3069,…
# $ title            <chr> "Large 5 Bed | 2 Baths 1st Floor Apartment - 10 Cente…
# $ full_addr        <chr> "10 Centennial Ave, Revere, MA 02151", "10 Independen…
# $ lat              <dbl> 42.40550, 42.05982, 42.20611, 42.43790, 42.43042, NA,…
# $ long             <dbl> -70.99678, -71.39399, -71.20505, -71.42923, -71.09099…
# $ query            <chr> "census", "census", "census", "census", "census", "",…
# $ dist_err         <dbl> 1154.073084, 3866.863403, 2149.685047, 4948.595847, 2…

Notice all the missing addresses with ‘high-accuracy’. Clearly the text address is an incomplete data. However, can you trust the CL created longitude and latitude?

library(ggplot2)

ggplot() +
  geom_point(aes(x = `data-accuracy`, y = `dist_err`), data = craigslist_table) +
  labs(x = "Data Accuracy", y = "Distance Error")

Exercise

• What can you infer from this graph? How does this condition your analysis further?

Hint: metres or kilometres? Compare with the size of a tract.

flag listings whose accuracy is poor or whose two locations disagree by more than a threshold
pick one location per listing, with a rule for which source you trust when
assign listings to tracts with each source and compare the counts

Now we are ready to combine them all into one table and visualise the results.

library(tmap.mapgl)   # adds the "maplibre" mode, which draws vector basemaps such as OpenFreeMap
tmap_mode("maplibre")

craigslist_sf <- craigslist_table %>%
  filter(!is.na(`data-longitude`)) %>%
  st_as_sf(coords = c("data-longitude", "data-latitude"), crs = 4326)

m1 <-
  tm_shape(craigslist_sf) +
  tm_dots(fill  = "prices",
             id = "titles",
             size = .3,
             popup = tm_popup(vars = c("prices", "full_addr")),
             group = "Rental listings") +
  tm_basemap("ofm.positron", group = "OpenFreeMap basemap")


library(widgetframe)
frameWidget(tmap_maplibre(m1))

Navigating to next page

The above code only extracts the listings that were on the page when we grabbed it. Craigslist no longer splits the results into numbered pages with a “next” button. Instead it shows one long, “pageless” list and fills it in as you scroll. To save memory, the browser keeps only about 200 listings in the page at any time and swaps batches in and out as you scroll up and down. So a single snapshot of the page source misses the rest of the results.

The page tells us how many listings there are in total. It is in the counter at the top of the results (e.g. “1 - 6 of 216”).

remDr$navigate(query_url)
Sys.sleep(3)

(total_count <- read_html(remDr$getPageSource()[[1]]) %>%
  html_element(".visible-counts") %>%
  html_text() %>%
  str_extract("(?<=of )[\\d,]+") %>%
  parse_number())
# [1] 216

To get all of them, we use Selenium to scroll down the page a screenful at a time, just like a person would. After each scroll we take the page source, parse the listings that are in it and add them to a table. Because the batches overlap, the same listing shows up many times, so we keep only one row per data-pid. We stop when we reach the bottom of the page and a few scrolls in a row have not turned up any new listings.

First, let’s wrap the parsing code from above into a function, so that we can reuse it on every snapshot.

parse_ads <- function(raw_ads){
  ads <- tibble(
    ids = html_attr(raw_ads, "data-pid"),
    titles = raw_ads %>% html_element("a.posting-title") %>% html_text(),
    prices = raw_ads %>% html_element("span.priceinfo") %>% html_text() %>% parse_number(),
    metadata = raw_ads %>% html_element("div.meta") %>% html_text(),
    urls = raw_ads %>% html_element(".cl-app-anchor") %>% html_attr("href")
  )
  return(ads)
}

Then scroll and collect.

all_ads <- tibble()
scroll_y <- 0
no_new_rounds <- 0

while (no_new_rounds < 3) {
  raw_ads <- read_html(remDr$getPageSource()[[1]]) %>%
    html_elements("div.cl-search-result")

  new_ads <- parse_ads(raw_ads) %>% filter(!ids %in% all_ads$ids)
  all_ads <- bind_rows(all_ads, new_ads)

  # Where are we on the page?
  page_height <- remDr$executeScript("return document.body.scrollHeight;")[[1]]
  at_bottom <- scroll_y >= page_height

  if (nrow(new_ads) == 0 & at_bottom) {
    no_new_rounds <- no_new_rounds + 1
  } else {
    no_new_rounds <- 0
  }

  # Scroll down by about a screenful and give the page time to load the next batch
  scroll_y <- scroll_y + 1500
  remDr$executeScript(paste0("window.scrollTo(0, ", scroll_y, ");"))
  Sys.sleep(1.5)
}

nrow(all_ads)
# [1] 217
all_ads %>% head()
# # A tibble: 6 × 5
#   ids        titles                                        prices metadata urls 
#   <chr>      <chr>                                          <dbl> <chr>    <chr>
# 1 7992711240 November 1st 2026 - 6 Bed 3 Bath  Pkg, Laund…   6700 30/106b… http…
# 2 7992710865 November 1st 2026 Cleveland Cir. 4 Bed, 2 Ba…   4500 30/104b… http…
# 3 7992506475 2 full baths: Updated kitchen and baths: one…   2800 30/102b… http…
# 4 7992455433 Large 5 Bed | 2 Baths 1st Floor Apartment - …   3500 30/105b… http…
# 5 7992381123 NEWLY ON 1 Family doesn't fail 2 impress. 4-…   4400 30/104b… http…
# 6 7992232695 Beautiful - 4 bedrooms,  2  baths, townhouse…   3550 29/104b… http…

We collected 217 listings, which you can compare with the count of 216 that the page reported. These may be off by a few, because new ads get posted and old ones get deleted while we are scrolling.

Note that we are using scrollTo with a position that we keep track of, rather than scrolling relative to where the page currently is. When the browser swaps batches of listings, the page can jump around, and an absolute position makes sure that we keep moving down.

Exercise

• Wrap the scrolling loop into a function that takes a query_url and returns the table of listings. Use it on a few other urban areas, such as Los Angeles and Raleigh. Notice that Craigslist redirects you to a slightly different url (remDr$getCurrentUrl()). Can you figure out how the query arguments were renamed?

• The more quickly you ping the server with your queries, you will run afoul of rate limits and may raise some eyebrows in San Francisco. Use Sys.sleep to slow down the query rate, especially when you then visit each individual ad for its location.

• Parse the metadata column in all_ads and plot the rent against the square footage.

• Do not forget to close your webdriver with remDr$close() when you are exiting the script.

Hint: one input, query_url. Compare the url you asked for with where you land.

function(query_url):
    open the url and wait
    start with an empty table and the scroll position at the top
    loop until the stopping rule holds:
        add the ads on the page that are not already in the table
        scroll down and wait
    return the table

for each city:
    build its query url, call the function, add a city column
    pause before the next city
stack the tables

pull the number in front of each unit out of the metadata text (NA if missing)
plot rent against size, coloured by bedrooms; try log scales

Cautions & Conclusions

Scraping the web to create a structured dataset is as much an art as it is a technical exercise. It is about figuring out the right combinations of searches and manipulations that will get you right result. And because there are no documentation and manuals, it is imperative to experiment with it and fail. And you will fail often. The webpages change their structure over time and the code that might work one week may not work the next. The webpages themselves are dynamic and therefore the dataset that you generate may not be replicable. All these should be taken into account when you are scraping the web for data and conducting your analyses.