Teadmised first lab 2012: erinevus redaktsioonide vahel
Resümee puudub |
Resümee puudub |
||
| 1. rida: | 1. rida: | ||
The main goal of the first lab is to web-scrape uncertain data from the public sources using search engines and | The main goal of the first lab is to web-scrape uncertain data from the public sources using search engines and calculate popularity index. | ||
== Conditions == | == Conditions == | ||
| 9. rida: | 9. rida: | ||
== What should the app do == | == What should the app do == | ||
Your system should take as input a list of arbitrary | Your system should take as input a list of arbitrary list of cities like "London", "New York", "Tallinn" etc (at least 10 cities) and use web search and web-scraping to obtain probabilistic and popularity data about the objects. We look for | ||
* | * population | ||
* | * area | ||
* search results (popularity) by different search engines (google, bing) in different search categories (web, images, videos, news) | |||
The data should be | The data should be enriched by a numeric confidence measure (0..1) indicating rough trust in the result. The confidence is mostly based on the sourcecpercentage of investigated pages/sources giving this result, as well as a perceived quality of the data source (like: wikipedia has high quality, arbitrary pages low). In order to get the list of pages to web scrape you should do several searches, selecting from | ||
* wikipedia, wolfram alpha search | |||
* google, bing (web, images, videos, news) web search | |||
The results should be scraped using suitable scraping method and confidence values should be set or calculated depending on the source or meaning of search query. All the captured properties and the confidence values should be used for popularity calculation and list of cities should be ordered by popularity. What is the mean of the calculated popularity? | |||
== Hints and suggestions == | == Hints and suggestions == | ||
| 32. rida: | 26. rida: | ||
While developing, it is easier to use a pre-stored search result/wikipedia page and pre-stored set of pages instead of doing full search and download each time. | While developing, it is easier to use a pre-stored search result/wikipedia page and pre-stored set of pages instead of doing full search and download each time. | ||
However, the system should finally work with | However, the system should finally work with custom place names, meaning that the pre-loaded data cannot be normally used. | ||
Interesting links and ideas: sophisticated systems. | Interesting links and ideas: sophisticated systems. | ||
Scraping frameworks: | |||
* [http://scraperwiki.com/ scraperite üldportaal] | |||
* [http://scrapy.org scrapy (python)] | |||
* [http://web-harvest.sourceforge.net/ wb-harvest (java)] | |||
Nell: | Nell: | ||
| 40. rida: | 39. rida: | ||
* [http://www.cs.cmu.edu/~rcwang/ Richard Wang] | * [http://www.cs.cmu.edu/~rcwang/ Richard Wang] | ||
* [http://www.boowa.com/ boowa extractor] | * [http://www.boowa.com/ boowa extractor] | ||
Peter Norvigi | Peter Norvigi | ||
[http://www.nypost.com/p/news/opinion/opedcolumnists/the_machine_age_tM7xPAv4pI4JslK0M1JtxI/2 artikkel statistikapõhisest rakenduslikust tehismõistusest] | [http://www.nypost.com/p/news/opinion/opedcolumnists/the_machine_age_tM7xPAv4pI4JslK0M1JtxI/2 artikkel statistikapõhisest rakenduslikust tehismõistusest] | ||
Other: | Other: | ||
| 55. rida: | 48. rida: | ||
* http://www.cs.uic.edu/~liub/teach/cs511-spring-06/CS511-opinion-synthesis.ppt | * http://www.cs.uic.edu/~liub/teach/cs511-spring-06/CS511-opinion-synthesis.ppt | ||
* http://www.cato.org/pubs/pas/pa584.pdf | * http://www.cato.org/pubs/pas/pa584.pdf | ||
Viimane redaktsioon: 20. veebruar 2012, kell 19:31
The main goal of the first lab is to web-scrape uncertain data from the public sources using search engines and calculate popularity index.
Conditions
You can do the lab either alone or together with a fellow student. Three students per lab is not ok, though.
Technology - language, op system etc - is completely free.
What should the app do
Your system should take as input a list of arbitrary list of cities like "London", "New York", "Tallinn" etc (at least 10 cities) and use web search and web-scraping to obtain probabilistic and popularity data about the objects. We look for
- population
- area
- search results (popularity) by different search engines (google, bing) in different search categories (web, images, videos, news)
The data should be enriched by a numeric confidence measure (0..1) indicating rough trust in the result. The confidence is mostly based on the sourcecpercentage of investigated pages/sources giving this result, as well as a perceived quality of the data source (like: wikipedia has high quality, arbitrary pages low). In order to get the list of pages to web scrape you should do several searches, selecting from
- wikipedia, wolfram alpha search
- google, bing (web, images, videos, news) web search
The results should be scraped using suitable scraping method and confidence values should be set or calculated depending on the source or meaning of search query. All the captured properties and the confidence values should be used for popularity calculation and list of cities should be ordered by popularity. What is the mean of the calculated popularity?
Hints and suggestions
While developing, it is easier to use a pre-stored search result/wikipedia page and pre-stored set of pages instead of doing full search and download each time.
However, the system should finally work with custom place names, meaning that the pre-loaded data cannot be normally used.
Interesting links and ideas: sophisticated systems.
Scraping frameworks:
Nell:
Peter Norvigi artikkel statistikapõhisest rakenduslikust tehismõistusest
Other: