Andme-apide märkmed loengust

General

Programmable web is a great site giving pointers to a large set of open data apis people have created.

Use curl to try the examples out or use the small python code down below to ease experimenting

Background example

See sightsmap as an example of a system based on a collection of large datasets obtained from open apis.

Also read how the sightsmap system is built

Weather

Weather underground example http://api.wunderground.com/api/da9972c7aa341e74/conditions/q/CA/Tartu.json

EMHI example http://www.emhi.ee/ilma_andmed/xml/observations.php

See emhi docs

Panoramio photos

Example: http://www.panoramio.com/map/get_panoramas.php?set=full&size=original&from=0&to=99&minx=24.73&miny=59.428&maxx=24.76&maxy=59.445&mapfilter=false

See panoramio api docs

Foursquare

Example (replace MYTOKEN, you can make it on the docs site or by going to https://developer.foursquare.com/docs/explore#req=users/requests)

https://api.foursquare.com/v2/venues/search?ll=59.437,24.745&radius=1000&&oauth_token=MYTOKEN&v=20141001&limit=50&intent=browse

See foursquare dev docs

Google places

Example (replace MYKEY, you can make it on the docs site) https://maps.googleapis.com/maps/api/place/search/json?location=-33.8670522,151.1957362&radius=500&types=food&name=harbour&sensor=false&key=MYKEY

Twitter

Example url from old api, which is no longer functional: http://search.twitter.com/search.json?q=%40tallinn

See twitter dev docs and twitter search docs

API testing console: https://dev.twitter.com/rest/tools/console

Google search

Example (replace MYKEY, you can make it on the docs site) https://www.googleapis.com/customsearch/v1?key=MYKEY&q=lectures

See web search docs and custom-search docs

Facebook graph api

https://developers.facebook.com/docs/graph-api

Quick example: https://graph.facebook.com/me?fields=id,name&access_token=MY_TOKEN

you can get MY_TOKEN from several places, for example: https://developers.facebook.com/tools/explorer?method=GET&path=me%3Ffields%3Did%2Cname&version=v2.1

Check also non-api scraping:

• http://www.pcworld.com/article/2056080/facebook-stalker-tool-uses-graph-search-for-powerful-data-mining.html
• https://github.com/milo2012/osintstalker

Python example for experimenting

#!/usr/bin/python

from math import *
import os
import urllib2
import json
import xml.dom.minidom

# NB! Replace YOURKEY and YOURTOKEN below with a key and token you make yourself
# on the corresponding api doc page

#url="http://api.wunderground.com/api/da9972c7aa341e74/conditions/q/CA/Tartu.json"
#url="http://www.panoramio.com/map/get_panoramas.php?set=full&size=original&from=0&to=99&minx=24.73&miny=59.428&maxx=24.76&maxy=59.445&mapfilter=false"
#url="https://api.foursquare.com/v2/venues/search?ll=59.437,24.745&radius=1000&&oauth_token=YOURTOKEN&v=20120103&limit=50&intent=browse"
#url="https://maps.googleapis.com/maps/api/place/search/json?location=-33.8670522,151.1957362&radius=500&sensor=false&key=YOURKEY"
url="http://search.twitter.com/search.json?q=tallinn"
#url="https://www.googleapis.com/customsearch/v1?key=YOURKEY&q=lectures"

def main():
  f=urllib2.urlopen(url)
  s=f.read()
  data=json.loads(s)
  #print data
  print json.dumps(data, sort_keys=True, indent=2)
  

def main2():
  req=urllib2.Request(url=url)
  rawresp=urllib2.urlopen(req)
  print "headers:\n"
  h=rawresp.info().headers
  for e in h: print e,
  s=rawresp.read()    
  data=json.loads(s)
  print "\ndata:\n"
  print json.dumps(data, sort_keys=True, indent=2)

  
main2()  

Microsoft Open Data web api standard

See

• Front page: http://www.odata.org
• First tutorial: http://www.odata.org/getting-started/basic-tutorial/
• Quick crucial docs: http://docs.oasis-open.org/odata/odata/v4.0/odata-v4.0-part2-url-conventions.html