Anient OG X Kaya's Dream (Ruby Slipper)
Primal Hemp Wrap
Cool Blue Gatorade

Searching craigslist takes too long

It would be nice to see rentals without having to actually click through...

One of the challenges of living in San Francisco is housing. The better places to rent from are usually gone by the time I've reached out for a viewing. So I came up with a way to send filtered results to a dedicated Slack channel, this way I see only what matters to me all day. In less than one week I'd found a great place to live for a while.

Requirements

  • Slack API Token
  • python-craigslist
    pip install python-craigslist==1.0.4
  • slackclient
    pip install slackclient==1.0.5
  • SLACK_TOKEN environment variable
    export SLACK_TOKEN=<place-your-slack-token-here>

Craigslist

from __future__ import unicode_literals

from craigslist import CraigslistHousing

rental_ids = set()


def find_housing(price='2500', location='', cat='hhh', private=True):
    rentals = CraigslistHousing(site='sfbay', area=location, category=cat,
                                filters={'max_price': price, 'private_room': private})
    houses = rentals.get_results(sort_by='newest', geotagged=True)
    count = 0
    responses = []
    for house in houses:
        res_map = {
            "name": house['name'] if house['name'] else '',
            "url": house['url'] if house['url'] else '',
            "price": house['price'] if house['price'] else '',
            "location": house['where'] if house['where'] else '',
        }
        rental_id = filter(lambda x: x.isdigit(), res_map['url'])
        if rental_id not in rental_ids:
            rental_ids.add(int(rental_id))
            bot_response = {
                "attachments": [
                    {
                        "fallback": "Craigslist SF",
                        "color": "#36a64f",
                        "title": res_map['name'],
                        "title_link": res_map['url'],
                        "text": res_map['price'],
                        "fields": [
                            {
                                "title": res_map['location']
                            }
                        ],
                        "footer": "Craigslist"}
                ]
            }
            responses.append(bot_response)
            count += 1
            if count > 25: break
    return responses

The script returns 26 unique results in the form of a Slack message attachment. Take a look at Slack's message formatting documentation to see how attachments work.

Integrate with Slack

I like wrapping code into classes. Here is how I wrapped the script from above into a class along with the Slack integration:

from __future__ import unicode_literals

import os

from craigslist import CraigslistHousing
from slackclient import SlackClient


class CLBot(object):

    def __init__(self):
        self.slack = SlackClient(os.environ.get('SLACK_TOKEN'))
        self.rental_ids = set()

    def post_message(self, channel, text, username='CLBOT'):
        self.slack.api_call("chat.postMessage", channel=channel, text=text, username=username, unfurl_links="true")

    def find_housing(self, price='2500', location='', cat='hhh', private=True):
        rentals = CraigslistHousing(site='sfbay', area=location, category=cat,
                                    filters={'max_price': price, 'private_room': private})
        houses = rentals.get_results(sort_by='newest', geotagged=True)
        count = 0
        responses = []
        for house in houses:
            res_map = {
                "name": house['name'] if house['name'] else '',
                "url": house['url'] if house['url'] else '',
                "price": house['price'] if house['price'] else '',
                "location": house['where'] if house['where'] else '',
            }
            rental_id = filter(lambda x: x.isdigit(), res_map['url'])
            if rental_id not in self.rental_ids:
                self.rental_ids.add(int(rental_id))
                bot_response = {
                    "attachments": [
                        {
                            "fallback": "Craigslist SF",
                            "color": "#36a64f",
                            "title": res_map['name'],
                            "title_link": res_map['url'],
                            "text": res_map['price'],
                            "fields": [
                                {
                                    "title": res_map['location']
                                }
                            ],
                            "footer": "Craigslist"}
                    ]
                }
                responses.append(bot_response)
                count += 1
                if count > 25: break
        return responses

You may have to create a dedicated Slack channel to send results. I find it easier using string-format for channel name, but you can also use channel ID if you know it (requires using Slack API to list all channels)

We can test results by using a python the interpreter. I personally prefer iPython

In [32]: bot = CLBot()

In [33]: bot.find_housing()
Out[33]:
[{u'attachments': [{u'color': u'#36a64f',
    u'fallback': u'Craigslist SF',
    u'fields': [{u'title': u'richmond / point / annex'}],
    u'footer': u'Craigslist',
    u'text': u'$499',
    u'title': u'Nice clean Room for rent shared in a 2 bed room apartment',
    u'title_link': u'https://sfbay.craigslist.org/eby/roo/d/richmond-nice-clean-room-for-rent/6809782837.html'}]},
 {u'attachments': [{u'color': u'#36a64f',

Now I simply add a task to run by using Python's schedule module

if __name__ == '__main__':
    bot = CLBot()
    schedule.every(20).minutes.do(bot.reminder_find_housing, channel='craigslist_slack')
    while True:
        try:
            schedule.run_pending()
        except KeyboardInterrupt:
            sys.exit()


Full Script

from __future__ import unicode_literals

import os
import sys

import schedule
from craigslist import CraigslistHousing
from slackclient import SlackClient


class CLBot(object):

    def __init__(self):
        self.slack = SlackClient(os.environ.get('SLACK_TOKEN'))
        self.rental_ids = set()

    def post_message(self, channel, text, username='CLBOT'):
        self.slack.api_call("chat.postMessage", channel=channel, text=text, username=username, unfurl_links="true")

    def reminder_find_housing(self, channel=''):
        """
        channel: id or channel name
        reminder for craigslist housing
        :return:
        """
        if not channel:
            raise KeyError("Must include a channel or the notification wont send")
        rentals = self.find_housing()
        for rental in rentals:
            self.post_message(channel, rental)

    def find_housing(self, price='2500', location='', cat='hhh', private=True):
        rentals = CraigslistHousing(site='sfbay', area=location, category=cat,
                                    filters={'max_price': price, 'private_room': private})
        houses = rentals.get_results(sort_by='newest', geotagged=True)
        count = 0
        responses = []
        for house in houses:
            res_map = {
                "name": house['name'] if house['name'] else '',
                "url": house['url'] if house['url'] else '',
                "price": house['price'] if house['price'] else '',
                "location": house['where'] if house['where'] else '',
            }
            rental_id = filter(lambda x: x.isdigit(), res_map['url'])
            if rental_id not in self.rental_ids:
                self.rental_ids.add(int(rental_id))
                bot_response = {
                    "attachments": [
                        {
                            "fallback": "Craigslist SF",
                            "color": "#36a64f",
                            "title": res_map['name'],
                            "title_link": res_map['url'],
                            "text": res_map['price'],
                            "fields": [
                                {
                                    "title": res_map['location']
                                }
                            ],
                            "footer": "Craigslist"}
                    ]
                }
                responses.append(bot_response)
                count += 1
                if count > 25: break
        return responses


if __name__ == '__main__':
    bot = CLBot()
    schedule.every(20).minutes.do(bot.reminder_find_housing, channel='craigslist_slack')
    while True:
        try:
            schedule.run_pending()
        except KeyboardInterrupt:
            sys.exit()

Then I deployed to Heroku

Slack Craigslist Bot