Lambda - Kasutaja kaastöö [et]

Teadmiste otsing, formaliseerimine ja hoidmine

2012-02-20T17:36:39Z

Kalle:

<table>
<tr>
<td>
[[Pilt:Mephistophiles small.gif| left|teadmised]]
</td>
<td>
Code: ITV0060 
Link: http://www.lambda.ee/index.php/Teadmiste_otsing,_formaliseerimine_ja_hoidmine or http://www.lambda.ee/index/itv0060 
Lecturer: Kalle Tomingas, Tanel Tammet 
Contact: kalle.tomingas@gmail.com, 50 40 568 
tammet@staff.ttu.ee, 6203457, TTÜ AK223 
Archives of previous years: [[itv0060 2011 arhiiv|2011]], [[itv0060 20010 arhiiv|2010]], [[itv0060 2009 arhiiv|2009]], [[itv0060 2008 arhiiv|2008]], [[itv0060 2007 arhiiv|2007]], [[itv0060 2006 arhiiv|2006]], [http:/cs.ttu.ee/kursused/wav4130/ older].
</td>
</tr>
</table>

== Time, place, result ==

Semester: spring 
Grading: exam 
Points: 3.5 

Lectures: every Monday 17.45-19.15, room IT-137a 
Practical work: Mondays on odd weeks 19.15-20.00, room IT-137a: 6 feb, 20 feb, 5 mar, 19 mar, 2 apr, 16 apr, 30 apr, 14 may, 21 may.

Practical work will give 40% and exam 60% of points underlying the final grade.

== Practical work ==

There are two labs on the same theme and an optional third lab.

=== First lab ===

The main goal of the first lab is to web-scrape data from the public sources using search engines and simple text analysis/statistics.

Details in [[Teadmised first lab 2012|Teadmised first lab 2012]]. The lab should be completed by March, later presentation will cut the points to half.

=== Second lab ===

The main goal of the second lab is to augment data obtained from the first lab by writing and using probabilistic rules with a rule engine.

Details about second lab will be available on beginning of April. Previous year lab can be seen here: [[Teadmised second lab 2011]].

=== Optional third lab ===

The optional third lab can be done in parallel with the second. It will be a relatively complex lab: successful passing will give additional 20 points, thus the weight of the exam is diminished from 60% to 40% of the overall score.

What to do: use Attempto to create a system such that you can write rules and/or facts and/or queries for the second lab in restricted english. These restricted english sentences have to be automatically converted to the form usable by Otter. It is a good idea to avoid probabilities in these rules, since it would be very hard to encode probabilities using attempto.

See: http://attempto.ifi.uzh.ch/site/ , http://attempto.ifi.uzh.ch/ape/

== Materials to study for the exam ==

Eksamiküsimused:

Väikesed ülesanded: data ja reeglite
esitamine ja kasutamine

Andmebaas nö tüüpilise relatsioonilise
mudelina, key-value ja rdf kujul

Ülesanne blocks worldist: esita mingi
olukord, mingi reegel ja näita rakendamise
tulemust

RDF, RDFS ja OWL põhiasjad: väike
ülesanne. Süntaksid ja eridetailid
ei ole olulised.

RDF ja RDFS esitada loogikas
faktide ja reeglitena, mõni
lihtne OWLi näide võib ka olla

Arusaamine RDFa põhiasjadest, oskus
teha näidet.

MÕni üldküsimus Nelli ja Attempto
kohta.

Arusaam et mis on mittemonotoonne
loogika ja mis ei ole, mis oleks
viisid esitada vasturääkivaid
teadmisi. Võib olla väike ülesanne
mingite asjade esitamise kohta: näide
ja selgitus.

Default loogika ülesanne a la linnud
ja Quaker diamond: panna kirja reeglid
ja viia läbi järeldused ja vbl selgitada
lisaks. Seos CWA, Prolog ja default loogika
vahel võib samuti küsimuse alla tulla.

Indeksid: arusaam baasindekseerimis-ideedest
ja B+ treest. Ntx ehitada B+ mingi
väikese datahulga jaoks. Sama hashi kohta
ja bit indeksite kohta. Eriteemad a la
indeksi pakkimine ja erinevad hashi
viisid ei tule teemaks.

Täistekstiindeksid: osata ehitada näite
pealt. Vt presentatsioonist erinevaid
viise, ntx Patricia trie.

Termiindeksid: piisab discrimination
tree indeksist ja näite põhjal
indeksi ehitamisest.

=== Declarative and procedural representations ===

Lecture material:
* [[Media:Kr intro.ppt | Intro lecture: declarative and procedural representations]]

=== Databases: meaning and representation of facts ===

* [[Media:Kr_plain_data_db_logic.ppt | relation of plain data in databases to logic & representing complex structures in databases ]]
* [[Media:Kr_rdf_schemalessdb_3.pdf|key/value pairs and rdf, compared to the relational model]]

=== Rules in logic: apps in planning, robotics, ... ===

Lecture material:
* [[Media:Kr_rules1.pdf|Rules and logic: refresher]] and as [[Media:Kr_rules1.ppt|ppt]]

See also:

* [[Media:Geoffreasoningnotes.pdf|Geoff reasoning course notes: just for background, no questions in exam]]
* [http://www.ee.pdx.edu/~mperkows/May4/005.Ch11-planning.ppt planning and blocks world]
* [[blocks world axiom/query examples from lecture]]

=== RDF, RDFS, OWL and conversion to 1st order logic ===

Lecture material:
* [[Media:Kr_rdfs.pdf|RDFS: rdf schema]] and as [[Media:Kr_rdfs.ppt|ppt]]

Understand RDF and RDFS:

* [http://en.wikipedia.org/wiki/Resource_Description_Framework RDF wikipedia]
* [http://www.w3.org/TR/rdf-primer/ w3c RDF primer]

Understand basics of owl:

* [http://en.wikipedia.org/wiki/Web_Ontology_Language OWL wikipedia]
* [http://www.w3.org/TR/owl-guide/ w3c OWL guide]

=== RDFa, reading and using RDF data from web pages ===

Understand main parts:

* [http://en.wikipedia.org/wiki/RDFa wiki intro]
* [[Media:portaalidekoosvoime.ppt]] or as pdf [[Media:portaalidekoosvoime.pdf]]
* [http://www.w3.org/2006/07/SWD/RDFa/primer/ w3c rdfa primer]

=== RDF relation to logic ===

Notes from lecture:
* [[Notes about RDF and OWL: logical meaning]]

Understand a bit of:
* [http://www.w3.org/TR/2003/NOTE-lbase-20031010/ Hayes & Guha Lbase]

=== Semantics of ordinary relational base ===

* [http://cs.ttu.ee/kursused/wav4130/rdftutapp.pdf rdf tutorial addendum for logic and dbases]

=== First order logic in query engines and planning ===

* [[Media:Geoffreasoningnotes.pdf|Geoff reasoning course notes: just for background, no questions in exam]]
* [http://www.ee.pdx.edu/~mperkows/May4/005.Ch11-planning.ppt planning and blocks world]
* [[blocks world axiom/query examples from lecture]]

=== Data extraction ===

Web mining by Nell:
* [http://rtw.ml.cmu.edu/rtw/ Nell]

=== Restricted english jms ===

* [http://attempto.ifi.unizh.ch/site/pubs/papers/phd_kaljurand.pdf Attempto restricted english]

=== Uncertain knowledge logic ===

* [http://plato.stanford.edu/entries/logic-nonmonotonic/ nonmonotonic logic]
* [http://en.wikipedia.org/wiki/Default_logic default logic]
* [http://sc.snu.ac.kr/courses/2008/spring/pg/krr/Lectures/Lecture11/Lec11.ppt Reasoning with Conflicting Knowledge]

=== Indexes ===

Traditional database indexes incl B+ tree:
* [http://www.lambda.ee/index.php/Pilt:IndexNotes04.ppt G. Molina lecture 4]

Bitmap indexes:
* [http://en.wikipedia.org/wiki/Bitmap_index wiki]

Hash indexes:
* [http://en.wikipedia.org/wiki/Hash_table wiki]

Fulltext indexes:
* [http://www.lambda.ee/index.php/Pilt:Fulltext_indexes.ppt good intro and overview]

Term indexes:
Understand basics of discrimination tree trie-type term indexes and path indexes from
* [http://www.lambda.ee/index.php/Pilt:Mccuneindexing.pdf mccune paper]

== Course structure ==

We start with representing and converting facts, continue to rules, first monotonic, then probabilistic and nonmonotonic, then look into search and indexing of complex information, and finally, some learning.

=== Intro and paradigms of knowledge representation ===

Lecture materials:
* [[Media:Kr intro.ppt | Intro lecture: declarative and procedural representations]]
* [http://dl.dropbox.com/u/11794296/roboswarm_saso_2.pdf roboswarm example of combining dec and proc]

For KR, declarative and procedural rep see:
* [http://en.wikipedia.org/wiki/Knowledge_representation Short wiki overview of knowledge representation]
* [http://groups.csail.mit.edu/medg/ftp/psz/k-rep.html Long and philosophical MIT overview of knowledge representation]
* [http://www.cse.buffalo.edu/~rapaport/676/F01/proc.decl.html Pointers to classical papers]
* [http://www.cogarch.org/index.php/Main_Page A Survey of Cognitive and Agent Architectures]
* [ftp://reports.stanford.edu/pub/cstr/reports/cs/tr/76/581/CS-TR-76-581.pdf Classical KRL paper from 1977: Bobrow & Winograd]
Web mining by Nell:
* [http://rtw.ml.cmu.edu/rtw/ Nell]
* [http://www.cs.cmu.edu/~rcwang/ Richard Wang]
* [http://www.boowa.com/ boowa extractor]

=== Databases: meaning and representation of facts ===

Lecture material:


* [[Media:Kr_plain_data_db_logic.ppt | relation of plain data in databases to logic & representing complex structures in databases ]]

* [[Media:Kr_rdf_schemalessdb_3.pdf|key/value pairs and rdf, compared to the relational model]]

See also about databases and logic:

* [http://www.cs.rice.edu/~tlogic/Database/all-lectures.pdf Logic and Database Queries (readable)]
* [http://magenta.ruc.dk/dat_en/research/reports/127.pdf logic in databases workshop (complex stuff)]

See also about RDF and related:

* [http://cs.ttu.ee/kursused/wav4130/rdftutapp.pdf rdf tutorial addendum for logic and dbases]
* [http://en.wikipedia.org/wiki/Resource_Description_Framework RDF wikipedia]
* [http://www.w3.org/TR/rdf-primer/ w3c RDF primer]
* [http://en.wikipedia.org/wiki/RDFa wiki intro]
* [[Media:portaalidekoosvoime.ppt]] or as pdf [[Media:portaalidekoosvoime.pdf]]
* [http://www.w3.org/2006/07/SWD/RDFa/primer/ w3c rdfa primer]
* [http://tripletalk.wordpress.com/2011/01/25/rdfa-deployment-across-the-web/ rdfa and microformats deployment now]
* [http://www.microformats.org microformats: a similar technology]
* [http://www.w3.org/TR/2003/NOTE-lbase-20031010/ Hayes & Guha Lbase]

=== Rules in logic: apps in planning, robotics, ... ===

Lecture material:
* [[Media:Kr_rules1.pdf|Rules and logic: refresher]] and as [[Media:Kr_rules1.ppt|ppt]]

Topics:

* propositional, first order, higher order
* semantics/models
* proofs and deriving new information
* automated reasoning

See also:

* [[Media:Geoffreasoningnotes.pdf|Geoff reasoning course notes: just for background, no questions in exam]]
* [http://www.ee.pdx.edu/~mperkows/May4/005.Ch11-planning.ppt planning and blocks world]
* [[blocks world axiom/query examples from lecture]]

=== Various rule/logic languages ===

Lecture material:
* [[Media:Kr_rdfs.pdf|RDFS: rdf schema]] and as [[Media:Kr_rdfs.ppt|ppt]]
* [[Notes about RDF and OWL: logical meaning]]

Several languages:
* RDFS
* OWL
* KIF
* CL
* ontologies
* wordnet
* cyc
* restricted english systems

See also:
* [http://en.wikipedia.org/wiki/Resource_Description_Framework RDF wikipedia]
* [http://www.w3.org/TR/rdf-primer/ w3c RDF primer]
* [http://en.wikipedia.org/wiki/Web_Ontology_Language OWL wikipedia]
* [http://www.w3.org/TR/owl-guide/ w3c OWL guide]
* [http://cl.tamu.edu/ CL main page],
* [http://attempto.ifi.unizh.ch/site/pubs/papers/phd_kaljurand.pdf Attempto restricted english]

=== Scraping semistructured data ===

* Example for background: http://www.sightsmap.com/

=== Time, context, metainformation ===

=== Probabilistic and nonmonotonic reasoning ===

See also:
* [http://plato.stanford.edu/entries/logic-nonmonotonic/ nonmonotonic logic]
* [http://en.wikipedia.org/wiki/Default_logic default logic]

=== Personalized search and recommendations ===

=== Fast search of facts and rules using indexing ===

* [http://www.lambda.ee/index.php/Pilt:IndexNotes04.ppt G. Molina lecture 4]
* [http://www.lambda.ee/index.php/Pilt:IndexNotes05.ppt G. Molina lecture 5]
* [http://www.lambda.ee/index.php/Pilt:Fulltext_indexes.ppt good intro and overview]
* [http://www.lambda.ee/index.php/Pilt:Mccuneindexing.pdf mccune paper about term indexes]

=== Learning ===

See also:

* [http://www.cs.toronto.edu/~hinton/csc2515/notes/lec1.ppt intro]
* [http://gamma.cs.unc.edu/rob-course/public_html/LEC/MachineLearningMethods.ppt robots/cars]
* [http://www.cs.toronto.edu/~hinton/csc2515/notes/lec4.ppt backpropagation]
* [http://www.cs.toronto.edu/~hinton/csc2515/lectures.html good set of lecture materials]
* [http://www.cs.cmu.edu/~tom/mlbook-chapter-slides.html another good full set of lecture materials]
** sh [http://www.cs.cmu.edu/afs/cs.cmu.edu/project/theo-20/www/mlbook/ch4.pdf neural netw chapter]





[[Category:Kursus]]

Teadmised first lab 2012

2012-02-20T17:31:57Z

Kalle:

The main goal of the first lab is to web-scrape uncertain data from the public sources using search engines and calculate popularity index.

== Conditions ==

You can do the lab either alone or together with a fellow student. Three students per lab is not ok, though.

Technology - language, op system etc - is completely free.

== What should the app do ==

Your system should take as input a list of arbitrary list of cities like "London", "New York", "Tallinn" etc (at least 10 cities) and use web search and web-scraping to obtain probabilistic and popularity data about the objects. We look for

* population
* area
* search results (popularity) by different search engines (google, bing) in different search categories (web, images, videos, news)

The data should be enriched by a numeric confidence measure (0..1) indicating rough trust in the result. The confidence is mostly based on the sourcecpercentage of investigated pages/sources giving this result, as well as a perceived quality of the data source (like: wikipedia has high quality, arbitrary pages low). In order to get the list of pages to web scrape you should do several searches, selecting from
* wikipedia, wolfram alpha search
* google, bing (web, images, videos, news) web search

The results should be scraped using suitable scraping method and confidence values should be set or calculated depending on the source or meaning of search query. All the captured properties and the confidence values should be used for popularity calculation and list of cities should be ordered by popularity. What is the mean of the calculated popularity?

== Hints and suggestions ==

While developing, it is easier to use a pre-stored search result/wikipedia page and pre-stored set of pages instead of doing full search and download each time.

However, the system should finally work with custom place names, meaning that the pre-loaded data cannot be normally used.

Interesting links and ideas: sophisticated systems.

Scraping frameworks:
* [http://scraperwiki.com/ scraperite üldportaal]
* [http://scrapy.org scrapy (python)]
* [http://web-harvest.sourceforge.net/ wb-harvest (java)]

Nell:
* [http://rtw.ml.cmu.edu/rtw/ Nell]
* [http://www.cs.cmu.edu/~rcwang/ Richard Wang]
* [http://www.boowa.com/ boowa extractor]

Peter Norvigi
[http://www.nypost.com/p/news/opinion/opedcolumnists/the_machine_age_tM7xPAv4pI4JslK0M1JtxI/2 artikkel statistikapõhisest rakenduslikust tehismõistusest]

Other:
* http://www.cs.uic.edu/~liub/WebMiningBook.html
* http://www.frenchlane.com/Web-Content-Mining-4.pdf
* http://www.cs.uic.edu/~liub/teach/cs511-spring-06/CS511-opinion-synthesis.ppt
* http://www.cato.org/pubs/pas/pa584.pdf

Teadmiste otsing, formaliseerimine ja hoidmine

2012-02-20T16:27:20Z

Kalle:

<table>
<tr>
<td>
[[Pilt:Mephistophiles small.gif| left|teadmised]]
</td>
<td>
Code: ITV0060 
Link: http://www.lambda.ee/index.php/Teadmiste_otsing,_formaliseerimine_ja_hoidmine or http://www.lambda.ee/index/itv0060 
Lecturer: Kalle Tomingas, Tanel Tammet 
Contact: kalle.tomingas@gmail.com, 50 40 568 
tammet@staff.ttu.ee, 6203457, TTÜ AK223 
Archives of previous years: [[itv0060 2011 arhiiv|2011]], [[itv0060 20010 arhiiv|2010]], [[itv0060 2009 arhiiv|2009]], [[itv0060 2008 arhiiv|2008]], [[itv0060 2007 arhiiv|2007]], [[itv0060 2006 arhiiv|2006]], [http:/cs.ttu.ee/kursused/wav4130/ older].
</td>
</tr>
</table>

== Time, place, result ==

Semester: spring 
Grading: exam 
Points: 3.5 

Lectures: every Monday 17.45-19.15, room IT-137a 
Practical work: Mondays on odd weeks 19.15-20.00, room IT-137a: 6 feb, 20 feb, 5 mar, 19 mar, 2 apr, 16 apr, 30 apr, 14 may, 21 may.

Practical work will give 40% and exam 60% of points underlying the final grade.

== Practical work ==

There are two labs on the same theme and an optional third lab.

=== First lab ===

The main goal of the first lab is to web-scrape data from the public sources using search engines and simple text analysis/statistics.

Details in [[Teadmised first lab 2011|Teadmised first lab]]. The lab should be completed by March, later presentation will cut the points to half.

=== Second lab ===

The main goal of the second lab is to augment data obtained from the first lab by writing and using probabilistic rules with a rule engine.

Details in [[Teadmised second lab 2011]].

=== Optional third lab ===

The optional third lab can be done in parallel with the second. It will be a relatively complex lab: successful passing will give additional 20 points, thus the weight of the exam is diminished from 60% to 40% of the overall score.

What to do: use Attempto to create a system such that you can write rules and/or facts and/or queries for the second lab in restricted english. These restricted english sentences have to be automatically converted to the form usable by Otter. It is a good idea to avoid probabilities in these rules, since it would be very hard to encode probabilities using attempto.

See: http://attempto.ifi.uzh.ch/site/ , http://attempto.ifi.uzh.ch/ape/

== Materials to study for the exam ==

Eksamiküsimused:

Väikesed ülesanded: data ja reeglite
esitamine ja kasutamine

Andmebaas nö tüüpilise relatsioonilise
mudelina, key-value ja rdf kujul

Ülesanne blocks worldist: esita mingi
olukord, mingi reegel ja näita rakendamise
tulemust

RDF, RDFS ja OWL põhiasjad: väike
ülesanne. Süntaksid ja eridetailid
ei ole olulised.

RDF ja RDFS esitada loogikas
faktide ja reeglitena, mõni
lihtne OWLi näide võib ka olla

Arusaamine RDFa põhiasjadest, oskus
teha näidet.

MÕni üldküsimus Nelli ja Attempto
kohta.

Arusaam et mis on mittemonotoonne
loogika ja mis ei ole, mis oleks
viisid esitada vasturääkivaid
teadmisi. Võib olla väike ülesanne
mingite asjade esitamise kohta: näide
ja selgitus.

Default loogika ülesanne a la linnud
ja Quaker diamond: panna kirja reeglid
ja viia läbi järeldused ja vbl selgitada
lisaks. Seos CWA, Prolog ja default loogika
vahel võib samuti küsimuse alla tulla.

Indeksid: arusaam baasindekseerimis-ideedest
ja B+ treest. Ntx ehitada B+ mingi
väikese datahulga jaoks. Sama hashi kohta
ja bit indeksite kohta. Eriteemad a la
indeksi pakkimine ja erinevad hashi
viisid ei tule teemaks.

Täistekstiindeksid: osata ehitada näite
pealt. Vt presentatsioonist erinevaid
viise, ntx Patricia trie.

Termiindeksid: piisab discrimination
tree indeksist ja näite põhjal
indeksi ehitamisest.

=== Declarative and procedural representations ===

Lecture material:
* [[Media:Kr intro.ppt | Intro lecture: declarative and procedural representations]]

=== Databases: meaning and representation of facts ===

* [[Media:Kr_plain_data_db_logic.ppt | relation of plain data in databases to logic & representing complex structures in databases ]]
* [[Media:Kr_rdf_schemalessdb_3.pdf|key/value pairs and rdf, compared to the relational model]]

=== Rules in logic: apps in planning, robotics, ... ===

Lecture material:
* [[Media:Kr_rules1.pdf|Rules and logic: refresher]] and as [[Media:Kr_rules1.ppt|ppt]]

See also:

* [[Media:Geoffreasoningnotes.pdf|Geoff reasoning course notes: just for background, no questions in exam]]
* [http://www.ee.pdx.edu/~mperkows/May4/005.Ch11-planning.ppt planning and blocks world]
* [[blocks world axiom/query examples from lecture]]

=== RDF, RDFS, OWL and conversion to 1st order logic ===

Lecture material:
* [[Media:Kr_rdfs.pdf|RDFS: rdf schema]] and as [[Media:Kr_rdfs.ppt|ppt]]

Understand RDF and RDFS:

* [http://en.wikipedia.org/wiki/Resource_Description_Framework RDF wikipedia]
* [http://www.w3.org/TR/rdf-primer/ w3c RDF primer]

Understand basics of owl:

* [http://en.wikipedia.org/wiki/Web_Ontology_Language OWL wikipedia]
* [http://www.w3.org/TR/owl-guide/ w3c OWL guide]

=== RDFa, reading and using RDF data from web pages ===

Understand main parts:

* [http://en.wikipedia.org/wiki/RDFa wiki intro]
* [[Media:portaalidekoosvoime.ppt]] or as pdf [[Media:portaalidekoosvoime.pdf]]
* [http://www.w3.org/2006/07/SWD/RDFa/primer/ w3c rdfa primer]

=== RDF relation to logic ===

Notes from lecture:
* [[Notes about RDF and OWL: logical meaning]]

Understand a bit of:
* [http://www.w3.org/TR/2003/NOTE-lbase-20031010/ Hayes & Guha Lbase]

=== Semantics of ordinary relational base ===

* [http://cs.ttu.ee/kursused/wav4130/rdftutapp.pdf rdf tutorial addendum for logic and dbases]

=== First order logic in query engines and planning ===

* [[Media:Geoffreasoningnotes.pdf|Geoff reasoning course notes: just for background, no questions in exam]]
* [http://www.ee.pdx.edu/~mperkows/May4/005.Ch11-planning.ppt planning and blocks world]
* [[blocks world axiom/query examples from lecture]]

=== Data extraction ===

Web mining by Nell:
* [http://rtw.ml.cmu.edu/rtw/ Nell]

=== Restricted english jms ===

* [http://attempto.ifi.unizh.ch/site/pubs/papers/phd_kaljurand.pdf Attempto restricted english]

=== Uncertain knowledge logic ===

* [http://plato.stanford.edu/entries/logic-nonmonotonic/ nonmonotonic logic]
* [http://en.wikipedia.org/wiki/Default_logic default logic]
* [http://sc.snu.ac.kr/courses/2008/spring/pg/krr/Lectures/Lecture11/Lec11.ppt Reasoning with Conflicting Knowledge]

=== Indexes ===

Traditional database indexes incl B+ tree:
* [http://www.lambda.ee/index.php/Pilt:IndexNotes04.ppt G. Molina lecture 4]

Bitmap indexes:
* [http://en.wikipedia.org/wiki/Bitmap_index wiki]

Hash indexes:
* [http://en.wikipedia.org/wiki/Hash_table wiki]

Fulltext indexes:
* [http://www.lambda.ee/index.php/Pilt:Fulltext_indexes.ppt good intro and overview]

Term indexes:
Understand basics of discrimination tree trie-type term indexes and path indexes from
* [http://www.lambda.ee/index.php/Pilt:Mccuneindexing.pdf mccune paper]

== Course structure ==

We start with representing and converting facts, continue to rules, first monotonic, then probabilistic and nonmonotonic, then look into search and indexing of complex information, and finally, some learning.

=== Intro and paradigms of knowledge representation ===

Lecture materials:
* [[Media:Kr intro.ppt | Intro lecture: declarative and procedural representations]]
* [http://dl.dropbox.com/u/11794296/roboswarm_saso_2.pdf roboswarm example of combining dec and proc]

For KR, declarative and procedural rep see:
* [http://en.wikipedia.org/wiki/Knowledge_representation Short wiki overview of knowledge representation]
* [http://groups.csail.mit.edu/medg/ftp/psz/k-rep.html Long and philosophical MIT overview of knowledge representation]
* [http://www.cse.buffalo.edu/~rapaport/676/F01/proc.decl.html Pointers to classical papers]
* [http://www.cogarch.org/index.php/Main_Page A Survey of Cognitive and Agent Architectures]
* [ftp://reports.stanford.edu/pub/cstr/reports/cs/tr/76/581/CS-TR-76-581.pdf Classical KRL paper from 1977: Bobrow & Winograd]
Web mining by Nell:
* [http://rtw.ml.cmu.edu/rtw/ Nell]
* [http://www.cs.cmu.edu/~rcwang/ Richard Wang]
* [http://www.boowa.com/ boowa extractor]

=== Databases: meaning and representation of facts ===

Lecture material:


* [[Media:Kr_plain_data_db_logic.ppt | relation of plain data in databases to logic & representing complex structures in databases ]]

* [[Media:Kr_rdf_schemalessdb_3.pdf|key/value pairs and rdf, compared to the relational model]]

See also about databases and logic:

* [http://www.cs.rice.edu/~tlogic/Database/all-lectures.pdf Logic and Database Queries (readable)]
* [http://magenta.ruc.dk/dat_en/research/reports/127.pdf logic in databases workshop (complex stuff)]

See also about RDF and related:

* [http://cs.ttu.ee/kursused/wav4130/rdftutapp.pdf rdf tutorial addendum for logic and dbases]
* [http://en.wikipedia.org/wiki/Resource_Description_Framework RDF wikipedia]
* [http://www.w3.org/TR/rdf-primer/ w3c RDF primer]
* [http://en.wikipedia.org/wiki/RDFa wiki intro]
* [[Media:portaalidekoosvoime.ppt]] or as pdf [[Media:portaalidekoosvoime.pdf]]
* [http://www.w3.org/2006/07/SWD/RDFa/primer/ w3c rdfa primer]
* [http://tripletalk.wordpress.com/2011/01/25/rdfa-deployment-across-the-web/ rdfa and microformats deployment now]
* [http://www.microformats.org microformats: a similar technology]
* [http://www.w3.org/TR/2003/NOTE-lbase-20031010/ Hayes & Guha Lbase]

=== Rules in logic: apps in planning, robotics, ... ===

Lecture material:
* [[Media:Kr_rules1.pdf|Rules and logic: refresher]] and as [[Media:Kr_rules1.ppt|ppt]]

Topics:

* propositional, first order, higher order
* semantics/models
* proofs and deriving new information
* automated reasoning

See also:

* [[Media:Geoffreasoningnotes.pdf|Geoff reasoning course notes: just for background, no questions in exam]]
* [http://www.ee.pdx.edu/~mperkows/May4/005.Ch11-planning.ppt planning and blocks world]
* [[blocks world axiom/query examples from lecture]]

=== Various rule/logic languages ===

Lecture material:
* [[Media:Kr_rdfs.pdf|RDFS: rdf schema]] and as [[Media:Kr_rdfs.ppt|ppt]]
* [[Notes about RDF and OWL: logical meaning]]

Several languages:
* RDFS
* OWL
* KIF
* CL
* ontologies
* wordnet
* cyc
* restricted english systems

See also:
* [http://en.wikipedia.org/wiki/Resource_Description_Framework RDF wikipedia]
* [http://www.w3.org/TR/rdf-primer/ w3c RDF primer]
* [http://en.wikipedia.org/wiki/Web_Ontology_Language OWL wikipedia]
* [http://www.w3.org/TR/owl-guide/ w3c OWL guide]
* [http://cl.tamu.edu/ CL main page],
* [http://attempto.ifi.unizh.ch/site/pubs/papers/phd_kaljurand.pdf Attempto restricted english]

=== Scraping semistructured data ===

* Example for background: http://www.sightsmap.com/

=== Time, context, metainformation ===

=== Probabilistic and nonmonotonic reasoning ===

See also:
* [http://plato.stanford.edu/entries/logic-nonmonotonic/ nonmonotonic logic]
* [http://en.wikipedia.org/wiki/Default_logic default logic]

=== Personalized search and recommendations ===

=== Fast search of facts and rules using indexing ===

* [http://www.lambda.ee/index.php/Pilt:IndexNotes04.ppt G. Molina lecture 4]
* [http://www.lambda.ee/index.php/Pilt:IndexNotes05.ppt G. Molina lecture 5]
* [http://www.lambda.ee/index.php/Pilt:Fulltext_indexes.ppt good intro and overview]
* [http://www.lambda.ee/index.php/Pilt:Mccuneindexing.pdf mccune paper about term indexes]

=== Learning ===

See also:

* [http://www.cs.toronto.edu/~hinton/csc2515/notes/lec1.ppt intro]
* [http://gamma.cs.unc.edu/rob-course/public_html/LEC/MachineLearningMethods.ppt robots/cars]
* [http://www.cs.toronto.edu/~hinton/csc2515/notes/lec4.ppt backpropagation]
* [http://www.cs.toronto.edu/~hinton/csc2515/lectures.html good set of lecture materials]
* [http://www.cs.cmu.edu/~tom/mlbook-chapter-slides.html another good full set of lecture materials]
** sh [http://www.cs.cmu.edu/afs/cs.cmu.edu/project/theo-20/www/mlbook/ch4.pdf neural netw chapter]





[[Category:Kursus]]

Teadmiste otsing, formaliseerimine ja hoidmine

2012-02-20T16:13:08Z

Kalle: /* First lab */

<table>
<tr>
<td>
[[Pilt:Mephistophiles small.gif| left|teadmised]]
</td>
<td>
Code: ITV0060 
Link: http://www.lambda.ee/index.php/Teadmiste_otsing,_formaliseerimine_ja_hoidmine or http://www.lambda.ee/index/itv0060 
Lecturer: Tanel Tammet 
Contact: tammet@staff.ttu.ee, 6203457, TTÜ AK223 
Archives of previous years: [[itv0060 2011 arhiiv|2011]], [[itv0060 20010 arhiiv|2010]], [[itv0060 2009 arhiiv|2009]], [[itv0060 2008 arhiiv|2008]], [[itv0060 2007 arhiiv|2007]], [[itv0060 2006 arhiiv|2006]], [http:/cs.ttu.ee/kursused/wav4130/ older].
</td>
</tr>
</table>

== Time, place, result ==

Semester: spring 
Grading: exam 
Points: 3.5 

Lectures: every Monday 12.00-13.30, room IT-137a 
Practical work: Mondays on odd weeks 10.00-11.30, room IT-213H: 31 jan, 14 feb, 28 feb, 14 mar, 28 mar, 4 apr, 18 apr, 2 may, 16 may.

Practical work will give 40% and exam 60% of points underlying the final grade.

== Järeleksam 2011 sügis ==

Järeleksam toimub 26 august (reede): kell 10 auditoorium IT-137a

== Eksam ja konsultatsioon ==

16. mai: eksamiks õppimise materjalide ülevaade ja fikseerimine.

Eksam:

järeleksam 15 juuni (kolmapäev): kell 9 auditoorium IT-137a

15 juuni tulemused:, eksamipunktid viidud 0-60 skaalale:
* HP eksam 52p + 40p ettekanded, hinne 5
* GJ eksam 31p + 32p ettekanded, hinne 2

3 juuni (reede): kell 9 auditoorium VI-121

3 juuni tulemused:, eksamipunktid viidud 0-60 skaalale:
* TT eksam 36p + 40p ettekanded, hinne 3
* RL eksam 40p + 40p ettekanded, hinne 4
* HP eksam 28p + 40p ettekanded, hinne 2
* VP eksam 22p + 40p ettekanded, hinne 2
* DA eksam 57p + 40p ettekanded, hinne 5

8 juuni (kolmapäev): kell 9 auditoorium VII-131

8 juuni tulemused:, eksamipunktid viidud 0-60 skaalale:
* KL eksam 60p + 40p ettekanded, hinne 5
* ML eksam 51p + 40p ettekanded, hinne 5
* ET eksam 48p + 30p ettekanded, hinne 3
* DF eksam 58p + 40p ettekanded, hinne 5

Järeleksam kevadel:

* 15 juuni (kolmapäev): kell 9 auditoorium IT-137a

== Practical work ==

There are two labs on the same theme and an optional third lab.

=== First lab ===

The main goal of the first lab is to web-scrape data from the public sources using search engines and simple text analysis/statistics.

Details in [[Teadmised first lab 2011|Teadmised first lab]]. The lab should be completed by March, later presentation will cut the points to half.

=== Second lab ===

The main goal of the second lab is to augment data obtained from the first lab by writing and using probabilistic rules with a rule engine.

Details in [[Teadmised second lab 2011]].

=== Optional third lab ===

The optional third lab can be done in parallel with the second. It will be a relatively complex lab: successful passing will give additional 20 points, thus the weight of the exam is diminished from 60% to 40% of the overall score.

What to do: use Attempto to create a system such that you can write rules and/or facts and/or queries for the second lab in restricted english. These restricted english sentences have to be automatically converted to the form usable by Otter. It is a good idea to avoid probabilities in these rules, since it would be very hard to encode probabilities using attempto.

See: http://attempto.ifi.uzh.ch/site/ , http://attempto.ifi.uzh.ch/ape/

== Materials to study for the exam ==

Eksamiküsimused:

Väikesed ülesanded: data ja reeglite
esitamine ja kasutamine

Andmebaas nö tüüpilise relatsioonilise
mudelina, key-value ja rdf kujul

Ülesanne blocks worldist: esita mingi
olukord, mingi reegel ja näita rakendamise
tulemust

RDF, RDFS ja OWL põhiasjad: väike
ülesanne. Süntaksid ja eridetailid
ei ole olulised.

RDF ja RDFS esitada loogikas
faktide ja reeglitena, mõni
lihtne OWLi näide võib ka olla

Arusaamine RDFa põhiasjadest, oskus
teha näidet.

MÕni üldküsimus Nelli ja Attempto
kohta.

Arusaam et mis on mittemonotoonne
loogika ja mis ei ole, mis oleks
viisid esitada vasturääkivaid
teadmisi. Võib olla väike ülesanne
mingite asjade esitamise kohta: näide
ja selgitus.

Default loogika ülesanne a la linnud
ja Quaker diamond: panna kirja reeglid
ja viia läbi järeldused ja vbl selgitada
lisaks. Seos CWA, Prolog ja default loogika
vahel võib samuti küsimuse alla tulla.

Indeksid: arusaam baasindekseerimis-ideedest
ja B+ treest. Ntx ehitada B+ mingi
väikese datahulga jaoks. Sama hashi kohta
ja bit indeksite kohta. Eriteemad a la
indeksi pakkimine ja erinevad hashi
viisid ei tule teemaks.

Täistekstiindeksid: osata ehitada näite
pealt. Vt presentatsioonist erinevaid
viise, ntx Patricia trie.

Termiindeksid: piisab discrimination
tree indeksist ja näite põhjal
indeksi ehitamisest.

=== Declarative and procedural representations ===

Lecture material:
* [[Media:Kr intro.ppt | Intro lecture: declarative and procedural representations]]

=== Databases: meaning and representation of facts ===

* [[Media:Kr_plain_data_db_logic.ppt | relation of plain data in databases to logic & representing complex structures in databases ]]
* [[Media:Kr_rdf_schemalessdb_3.pdf|key/value pairs and rdf, compared to the relational model]]

=== Rules in logic: apps in planning, robotics, ... ===

Lecture material:
* [[Media:Kr_rules1.pdf|Rules and logic: refresher]] and as [[Media:Kr_rules1.ppt|ppt]]

See also:

* [[Media:Geoffreasoningnotes.pdf|Geoff reasoning course notes: just for background, no questions in exam]]
* [http://www.ee.pdx.edu/~mperkows/May4/005.Ch11-planning.ppt planning and blocks world]
* [[blocks world axiom/query examples from lecture]]

=== RDF, RDFS, OWL and conversion to 1st order logic ===

Lecture material:
* [[Media:Kr_rdfs.pdf|RDFS: rdf schema]] and as [[Media:Kr_rdfs.ppt|ppt]]

Understand RDF and RDFS:

* [http://en.wikipedia.org/wiki/Resource_Description_Framework RDF wikipedia]
* [http://www.w3.org/TR/rdf-primer/ w3c RDF primer]

Understand basics of owl:

* [http://en.wikipedia.org/wiki/Web_Ontology_Language OWL wikipedia]
* [http://www.w3.org/TR/owl-guide/ w3c OWL guide]

=== RDFa, reading and using RDF data from web pages ===

Understand main parts:

* [http://en.wikipedia.org/wiki/RDFa wiki intro]
* [[Media:portaalidekoosvoime.ppt]] or as pdf [[Media:portaalidekoosvoime.pdf]]
* [http://www.w3.org/2006/07/SWD/RDFa/primer/ w3c rdfa primer]

=== RDF relation to logic ===

Notes from lecture:
* [[Notes about RDF and OWL: logical meaning]]

Understand a bit of:
* [http://www.w3.org/TR/2003/NOTE-lbase-20031010/ Hayes & Guha Lbase]

=== Semantics of ordinary relational base ===

* [http://cs.ttu.ee/kursused/wav4130/rdftutapp.pdf rdf tutorial addendum for logic and dbases]

=== First order logic in query engines and planning ===

* [[Media:Geoffreasoningnotes.pdf|Geoff reasoning course notes: just for background, no questions in exam]]
* [http://www.ee.pdx.edu/~mperkows/May4/005.Ch11-planning.ppt planning and blocks world]
* [[blocks world axiom/query examples from lecture]]

=== Data extraction ===

Web mining by Nell:
* [http://rtw.ml.cmu.edu/rtw/ Nell]

=== Restricted english jms ===

* [http://attempto.ifi.unizh.ch/site/pubs/papers/phd_kaljurand.pdf Attempto restricted english]

=== Uncertain knowledge logic ===

* [http://plato.stanford.edu/entries/logic-nonmonotonic/ nonmonotonic logic]
* [http://en.wikipedia.org/wiki/Default_logic default logic]
* [http://sc.snu.ac.kr/courses/2008/spring/pg/krr/Lectures/Lecture11/Lec11.ppt Reasoning with Conflicting Knowledge]

=== Indexes ===

Traditional database indexes incl B+ tree:
* [http://www.lambda.ee/index.php/Pilt:IndexNotes04.ppt G. Molina lecture 4]

Bitmap indexes:
* [http://en.wikipedia.org/wiki/Bitmap_index wiki]

Hash indexes:
* [http://en.wikipedia.org/wiki/Hash_table wiki]

Fulltext indexes:
* [http://www.lambda.ee/index.php/Pilt:Fulltext_indexes.ppt good intro and overview]

Term indexes:
Understand basics of discrimination tree trie-type term indexes and path indexes from
* [http://www.lambda.ee/index.php/Pilt:Mccuneindexing.pdf mccune paper]

== Course structure ==

We start with representing and converting facts, continue to rules, first monotonic, then probabilistic and nonmonotonic, then look into search and indexing of complex information, and finally, some learning.

=== Intro and paradigms of knowledge representation ===

Lecture materials:
* [[Media:Kr intro.ppt | Intro lecture: declarative and procedural representations]]
* [http://dl.dropbox.com/u/11794296/roboswarm_saso_2.pdf roboswarm example of combining dec and proc]

For KR, declarative and procedural rep see:
* [http://en.wikipedia.org/wiki/Knowledge_representation Short wiki overview of knowledge representation]
* [http://groups.csail.mit.edu/medg/ftp/psz/k-rep.html Long and philosophical MIT overview of knowledge representation]
* [http://www.cse.buffalo.edu/~rapaport/676/F01/proc.decl.html Pointers to classical papers]
* [http://www.cogarch.org/index.php/Main_Page A Survey of Cognitive and Agent Architectures]
* [ftp://reports.stanford.edu/pub/cstr/reports/cs/tr/76/581/CS-TR-76-581.pdf Classical KRL paper from 1977: Bobrow & Winograd]
Web mining by Nell:
* [http://rtw.ml.cmu.edu/rtw/ Nell]
* [http://www.cs.cmu.edu/~rcwang/ Richard Wang]
* [http://www.boowa.com/ boowa extractor]

=== Databases: meaning and representation of facts ===

Lecture material:


* [[Media:Kr_plain_data_db_logic.ppt | relation of plain data in databases to logic & representing complex structures in databases ]]

* [[Media:Kr_rdf_schemalessdb_3.pdf|key/value pairs and rdf, compared to the relational model]]

See also about databases and logic:

* [http://www.cs.rice.edu/~tlogic/Database/all-lectures.pdf Logic and Database Queries (readable)]
* [http://magenta.ruc.dk/dat_en/research/reports/127.pdf logic in databases workshop (complex stuff)]

See also about RDF and related:

* [http://cs.ttu.ee/kursused/wav4130/rdftutapp.pdf rdf tutorial addendum for logic and dbases]
* [http://en.wikipedia.org/wiki/Resource_Description_Framework RDF wikipedia]
* [http://www.w3.org/TR/rdf-primer/ w3c RDF primer]
* [http://en.wikipedia.org/wiki/RDFa wiki intro]
* [[Media:portaalidekoosvoime.ppt]] or as pdf [[Media:portaalidekoosvoime.pdf]]
* [http://www.w3.org/2006/07/SWD/RDFa/primer/ w3c rdfa primer]
* [http://tripletalk.wordpress.com/2011/01/25/rdfa-deployment-across-the-web/ rdfa and microformats deployment now]
* [http://www.microformats.org microformats: a similar technology]
* [http://www.w3.org/TR/2003/NOTE-lbase-20031010/ Hayes & Guha Lbase]

=== Rules in logic: apps in planning, robotics, ... ===

Lecture material:
* [[Media:Kr_rules1.pdf|Rules and logic: refresher]] and as [[Media:Kr_rules1.ppt|ppt]]

Topics:

* propositional, first order, higher order
* semantics/models
* proofs and deriving new information
* automated reasoning

See also:

* [[Media:Geoffreasoningnotes.pdf|Geoff reasoning course notes: just for background, no questions in exam]]
* [http://www.ee.pdx.edu/~mperkows/May4/005.Ch11-planning.ppt planning and blocks world]
* [[blocks world axiom/query examples from lecture]]

=== Various rule/logic languages ===

Lecture material:
* [[Media:Kr_rdfs.pdf|RDFS: rdf schema]] and as [[Media:Kr_rdfs.ppt|ppt]]
* [[Notes about RDF and OWL: logical meaning]]

Several languages:
* RDFS
* OWL
* KIF
* CL
* ontologies
* wordnet
* cyc
* restricted english systems

See also:
* [http://en.wikipedia.org/wiki/Resource_Description_Framework RDF wikipedia]
* [http://www.w3.org/TR/rdf-primer/ w3c RDF primer]
* [http://en.wikipedia.org/wiki/Web_Ontology_Language OWL wikipedia]
* [http://www.w3.org/TR/owl-guide/ w3c OWL guide]
* [http://cl.tamu.edu/ CL main page],
* [http://attempto.ifi.unizh.ch/site/pubs/papers/phd_kaljurand.pdf Attempto restricted english]

=== Scraping semistructured data ===

* Example for background: http://www.sightsmap.com/

=== Time, context, metainformation ===

=== Probabilistic and nonmonotonic reasoning ===

See also:
* [http://plato.stanford.edu/entries/logic-nonmonotonic/ nonmonotonic logic]
* [http://en.wikipedia.org/wiki/Default_logic default logic]

=== Personalized search and recommendations ===

=== Fast search of facts and rules using indexing ===

* [http://www.lambda.ee/index.php/Pilt:IndexNotes04.ppt G. Molina lecture 4]
* [http://www.lambda.ee/index.php/Pilt:IndexNotes05.ppt G. Molina lecture 5]
* [http://www.lambda.ee/index.php/Pilt:Fulltext_indexes.ppt good intro and overview]
* [http://www.lambda.ee/index.php/Pilt:Mccuneindexing.pdf mccune paper about term indexes]

=== Learning ===

See also:

* [http://www.cs.toronto.edu/~hinton/csc2515/notes/lec1.ppt intro]
* [http://gamma.cs.unc.edu/rob-course/public_html/LEC/MachineLearningMethods.ppt robots/cars]
* [http://www.cs.toronto.edu/~hinton/csc2515/notes/lec4.ppt backpropagation]
* [http://www.cs.toronto.edu/~hinton/csc2515/lectures.html good set of lecture materials]
* [http://www.cs.cmu.edu/~tom/mlbook-chapter-slides.html another good full set of lecture materials]
** sh [http://www.cs.cmu.edu/afs/cs.cmu.edu/project/theo-20/www/mlbook/ch4.pdf neural netw chapter]





[[Category:Kursus]]

Teadmised first lab 2012

2012-02-20T16:12:32Z

Kalle:

The main goal of the first lab is to web-scrape uncertain data from the public sources using search engines and simple text analysis/statistics.

== Conditions ==

You can do the lab either alone or together with a fellow student. Three students per lab is not ok, though.

Technology - language, op system etc - is completely free.

== What should the app do ==

Your system should take as input a list of arbitrary investigated objects like "Tower Bridge", "Statue of Liberty", "Tallinna raekoda", "Olde hansa", "Tallinn University of Technology", "Andrus Ansip", "Ford focus", "inception" and use web search and web-scraping to obtain probabilistic data about the objects. We look for

* popularity (search hits)
* object type (building, church, tower, landmark, person, car, ...)
* creation year (building year, birthdate, year of introduction to market, ...)
* other often-used characteristic words/tags (steeple, national, medieval, academic, peaminister, thriller, ...)

The data should be augmented by a numeric confidence measure (0....1) indicating rough trust in the result. The confidence is mostly based on the percentage of investigated pages/sources giving this result, as well as a perceived quality of the data source (like: wikipedia has high quality, arbitrary pages low).

In order to get the list of pages to web scrape you should do several searches, selecting from / using in parallel

* wikipedia search
* wolfram alpha search
* google or bing (or some other) web search

The results should be scraped. Use a different scraping method for semistructured data (wikipedia, wolfram alpha) and unstructured data (arbitrary web pages). For the latter it makes sense to split the page into sentences (just use period . as a separator) and look for sentences containing the investigated phrase. What other words does it contain? What numbers does it contain?

After scraping N pages for the same phrase, look for the most highly occurring type words, year numbers, other tags.

== Hints and suggestions ==

While developing, it is easier to use a pre-stored search result/wikipedia page and pre-stored set of pages instead of doing full search and download each time.

However, the system should finally work with any phrases, meaning that the pre-loaded data cannot be normally used.

Interesting links and ideas: sophisticated systems.

Nell:
* [http://rtw.ml.cmu.edu/rtw/ Nell]
* [http://www.cs.cmu.edu/~rcwang/ Richard Wang]
* [http://www.boowa.com/ boowa extractor]

Peter Norvigi
[http://www.nypost.com/p/news/opinion/opedcolumnists/the_machine_age_tM7xPAv4pI4JslK0M1JtxI/2 artikkel statistikapõhisest rakenduslikust tehismõistusest]

Scrapimisframeworke:
* [http://scraperwiki.com/ scraperite üldportaal]
* [http://scrapy.org scrapy (python)]
* [http://web-harvest.sourceforge.net/ wb-harvest (java)]

Other:
* http://www.cs.uic.edu/~liub/WebMiningBook.html
* http://www.frenchlane.com/Web-Content-Mining-4.pdf
* http://www.cs.uic.edu/~liub/teach/cs511-spring-06/CS511-opinion-synthesis.ppt
* http://www.cato.org/pubs/pas/pa584.pdf

== Kuidas alustada ==

Ükski neist soovitustest pole kohustuslik, aga aitavad, kui oled segaduses:

* Tuttav progekeel java/python

* Teha rakendus mis tombab alla google otsingu tulemuse. Kolm võimalikku varianti, parem eelista seda, mis neist tundub sulle lihtsam (alltoodud järjekord ongi tõenäoline lihtsuse järjekord):

** html: headerisse ntx firefoxi user agent a la

User-Agent: Mozilla/5.0 (Windows; U;
Windows NT 6.1; ru; rv:1.9.2b5)
Gecko/20091204 Firefox/3.6b5
** vana api
** uus api

* Kui google leht käes (mistahes apiga)siis tee proge osa mis otsib lehelt kõik viidatud lingid. Trüki kontrolliks välja.

* Seejärel tõmba järjest endale kõik viidatud lehed. Saveda pole vaja. Trüki kontrolliks välja.

* Lisa tõmbamise koodile lehe source töötlus:
** spliti punktide järgi.
** otsi laused sinu otsisõnaga.
** loe kokku eri sõnad lauses ja save globaalsesse baasi a la
car: 1
wheels: 2
jne
** iga lehe järel muutuvad sõnade arvud baasis suuremaks

* Seejärel läheb asi huvitamaks ja keerulisemaks, samas on sul juba tekkinud intuitsioon, kuidas edasi.

Teadmiste otsing, formaliseerimine ja hoidmine

2012-02-20T16:12:15Z

Kalle: /* First lab */

<table>
<tr>
<td>
[[Pilt:Mephistophiles small.gif| left|teadmised]]
</td>
<td>
Code: ITV0060 
Link: http://www.lambda.ee/index.php/Teadmiste_otsing,_formaliseerimine_ja_hoidmine or http://www.lambda.ee/index/itv0060 
Lecturer: Tanel Tammet 
Contact: tammet@staff.ttu.ee, 6203457, TTÜ AK223 
Archives of previous years: [[itv0060 2011 arhiiv|2011]], [[itv0060 20010 arhiiv|2010]], [[itv0060 2009 arhiiv|2009]], [[itv0060 2008 arhiiv|2008]], [[itv0060 2007 arhiiv|2007]], [[itv0060 2006 arhiiv|2006]], [http:/cs.ttu.ee/kursused/wav4130/ older].
</td>
</tr>
</table>

== Time, place, result ==

Semester: spring 
Grading: exam 
Points: 3.5 

Lectures: every Monday 12.00-13.30, room IT-137a 
Practical work: Mondays on odd weeks 10.00-11.30, room IT-213H: 31 jan, 14 feb, 28 feb, 14 mar, 28 mar, 4 apr, 18 apr, 2 may, 16 may.

Practical work will give 40% and exam 60% of points underlying the final grade.

== Järeleksam 2011 sügis ==

Järeleksam toimub 26 august (reede): kell 10 auditoorium IT-137a

== Eksam ja konsultatsioon ==

16. mai: eksamiks õppimise materjalide ülevaade ja fikseerimine.

Eksam:

järeleksam 15 juuni (kolmapäev): kell 9 auditoorium IT-137a

15 juuni tulemused:, eksamipunktid viidud 0-60 skaalale:
* HP eksam 52p + 40p ettekanded, hinne 5
* GJ eksam 31p + 32p ettekanded, hinne 2

3 juuni (reede): kell 9 auditoorium VI-121

3 juuni tulemused:, eksamipunktid viidud 0-60 skaalale:
* TT eksam 36p + 40p ettekanded, hinne 3
* RL eksam 40p + 40p ettekanded, hinne 4
* HP eksam 28p + 40p ettekanded, hinne 2
* VP eksam 22p + 40p ettekanded, hinne 2
* DA eksam 57p + 40p ettekanded, hinne 5

8 juuni (kolmapäev): kell 9 auditoorium VII-131

8 juuni tulemused:, eksamipunktid viidud 0-60 skaalale:
* KL eksam 60p + 40p ettekanded, hinne 5
* ML eksam 51p + 40p ettekanded, hinne 5
* ET eksam 48p + 30p ettekanded, hinne 3
* DF eksam 58p + 40p ettekanded, hinne 5

Järeleksam kevadel:

* 15 juuni (kolmapäev): kell 9 auditoorium IT-137a

== Practical work ==

There are two labs on the same theme and an optional third lab.

=== First lab ===

The main goal of the first lab is to web-scrape data from the public sources using search engines and simple text analysis/statistics.

Details in [[Teadmised first lab 2012|Teadmised first lab]]. The lab should be completed by March, later presentation will cut the points to half.

=== Second lab ===

The main goal of the second lab is to augment data obtained from the first lab by writing and using probabilistic rules with a rule engine.

Details in [[Teadmised second lab 2011]].

=== Optional third lab ===

The optional third lab can be done in parallel with the second. It will be a relatively complex lab: successful passing will give additional 20 points, thus the weight of the exam is diminished from 60% to 40% of the overall score.

What to do: use Attempto to create a system such that you can write rules and/or facts and/or queries for the second lab in restricted english. These restricted english sentences have to be automatically converted to the form usable by Otter. It is a good idea to avoid probabilities in these rules, since it would be very hard to encode probabilities using attempto.

See: http://attempto.ifi.uzh.ch/site/ , http://attempto.ifi.uzh.ch/ape/

== Materials to study for the exam ==

Eksamiküsimused:

Väikesed ülesanded: data ja reeglite
esitamine ja kasutamine

Andmebaas nö tüüpilise relatsioonilise
mudelina, key-value ja rdf kujul

Ülesanne blocks worldist: esita mingi
olukord, mingi reegel ja näita rakendamise
tulemust

RDF, RDFS ja OWL põhiasjad: väike
ülesanne. Süntaksid ja eridetailid
ei ole olulised.

RDF ja RDFS esitada loogikas
faktide ja reeglitena, mõni
lihtne OWLi näide võib ka olla

Arusaamine RDFa põhiasjadest, oskus
teha näidet.

MÕni üldküsimus Nelli ja Attempto
kohta.

Arusaam et mis on mittemonotoonne
loogika ja mis ei ole, mis oleks
viisid esitada vasturääkivaid
teadmisi. Võib olla väike ülesanne
mingite asjade esitamise kohta: näide
ja selgitus.

Default loogika ülesanne a la linnud
ja Quaker diamond: panna kirja reeglid
ja viia läbi järeldused ja vbl selgitada
lisaks. Seos CWA, Prolog ja default loogika
vahel võib samuti küsimuse alla tulla.

Indeksid: arusaam baasindekseerimis-ideedest
ja B+ treest. Ntx ehitada B+ mingi
väikese datahulga jaoks. Sama hashi kohta
ja bit indeksite kohta. Eriteemad a la
indeksi pakkimine ja erinevad hashi
viisid ei tule teemaks.

Täistekstiindeksid: osata ehitada näite
pealt. Vt presentatsioonist erinevaid
viise, ntx Patricia trie.

Termiindeksid: piisab discrimination
tree indeksist ja näite põhjal
indeksi ehitamisest.

=== Declarative and procedural representations ===

Lecture material:
* [[Media:Kr intro.ppt | Intro lecture: declarative and procedural representations]]

=== Databases: meaning and representation of facts ===

* [[Media:Kr_plain_data_db_logic.ppt | relation of plain data in databases to logic & representing complex structures in databases ]]
* [[Media:Kr_rdf_schemalessdb_3.pdf|key/value pairs and rdf, compared to the relational model]]

=== Rules in logic: apps in planning, robotics, ... ===

Lecture material:
* [[Media:Kr_rules1.pdf|Rules and logic: refresher]] and as [[Media:Kr_rules1.ppt|ppt]]

See also:

* [[Media:Geoffreasoningnotes.pdf|Geoff reasoning course notes: just for background, no questions in exam]]
* [http://www.ee.pdx.edu/~mperkows/May4/005.Ch11-planning.ppt planning and blocks world]
* [[blocks world axiom/query examples from lecture]]

=== RDF, RDFS, OWL and conversion to 1st order logic ===

Lecture material:
* [[Media:Kr_rdfs.pdf|RDFS: rdf schema]] and as [[Media:Kr_rdfs.ppt|ppt]]

Understand RDF and RDFS:

* [http://en.wikipedia.org/wiki/Resource_Description_Framework RDF wikipedia]
* [http://www.w3.org/TR/rdf-primer/ w3c RDF primer]

Understand basics of owl:

* [http://en.wikipedia.org/wiki/Web_Ontology_Language OWL wikipedia]
* [http://www.w3.org/TR/owl-guide/ w3c OWL guide]

=== RDFa, reading and using RDF data from web pages ===

Understand main parts:

* [http://en.wikipedia.org/wiki/RDFa wiki intro]
* [[Media:portaalidekoosvoime.ppt]] or as pdf [[Media:portaalidekoosvoime.pdf]]
* [http://www.w3.org/2006/07/SWD/RDFa/primer/ w3c rdfa primer]

=== RDF relation to logic ===

Notes from lecture:
* [[Notes about RDF and OWL: logical meaning]]

Understand a bit of:
* [http://www.w3.org/TR/2003/NOTE-lbase-20031010/ Hayes & Guha Lbase]

=== Semantics of ordinary relational base ===

* [http://cs.ttu.ee/kursused/wav4130/rdftutapp.pdf rdf tutorial addendum for logic and dbases]

=== First order logic in query engines and planning ===

* [[Media:Geoffreasoningnotes.pdf|Geoff reasoning course notes: just for background, no questions in exam]]
* [http://www.ee.pdx.edu/~mperkows/May4/005.Ch11-planning.ppt planning and blocks world]
* [[blocks world axiom/query examples from lecture]]

=== Data extraction ===

Web mining by Nell:
* [http://rtw.ml.cmu.edu/rtw/ Nell]

=== Restricted english jms ===

* [http://attempto.ifi.unizh.ch/site/pubs/papers/phd_kaljurand.pdf Attempto restricted english]

=== Uncertain knowledge logic ===

* [http://plato.stanford.edu/entries/logic-nonmonotonic/ nonmonotonic logic]
* [http://en.wikipedia.org/wiki/Default_logic default logic]
* [http://sc.snu.ac.kr/courses/2008/spring/pg/krr/Lectures/Lecture11/Lec11.ppt Reasoning with Conflicting Knowledge]

=== Indexes ===

Traditional database indexes incl B+ tree:
* [http://www.lambda.ee/index.php/Pilt:IndexNotes04.ppt G. Molina lecture 4]

Bitmap indexes:
* [http://en.wikipedia.org/wiki/Bitmap_index wiki]

Hash indexes:
* [http://en.wikipedia.org/wiki/Hash_table wiki]

Fulltext indexes:
* [http://www.lambda.ee/index.php/Pilt:Fulltext_indexes.ppt good intro and overview]

Term indexes:
Understand basics of discrimination tree trie-type term indexes and path indexes from
* [http://www.lambda.ee/index.php/Pilt:Mccuneindexing.pdf mccune paper]

== Course structure ==

We start with representing and converting facts, continue to rules, first monotonic, then probabilistic and nonmonotonic, then look into search and indexing of complex information, and finally, some learning.

=== Intro and paradigms of knowledge representation ===

Lecture materials:
* [[Media:Kr intro.ppt | Intro lecture: declarative and procedural representations]]
* [http://dl.dropbox.com/u/11794296/roboswarm_saso_2.pdf roboswarm example of combining dec and proc]

For KR, declarative and procedural rep see:
* [http://en.wikipedia.org/wiki/Knowledge_representation Short wiki overview of knowledge representation]
* [http://groups.csail.mit.edu/medg/ftp/psz/k-rep.html Long and philosophical MIT overview of knowledge representation]
* [http://www.cse.buffalo.edu/~rapaport/676/F01/proc.decl.html Pointers to classical papers]
* [http://www.cogarch.org/index.php/Main_Page A Survey of Cognitive and Agent Architectures]
* [ftp://reports.stanford.edu/pub/cstr/reports/cs/tr/76/581/CS-TR-76-581.pdf Classical KRL paper from 1977: Bobrow & Winograd]
Web mining by Nell:
* [http://rtw.ml.cmu.edu/rtw/ Nell]
* [http://www.cs.cmu.edu/~rcwang/ Richard Wang]
* [http://www.boowa.com/ boowa extractor]

=== Databases: meaning and representation of facts ===

Lecture material:


* [[Media:Kr_plain_data_db_logic.ppt | relation of plain data in databases to logic & representing complex structures in databases ]]

* [[Media:Kr_rdf_schemalessdb_3.pdf|key/value pairs and rdf, compared to the relational model]]

See also about databases and logic:

* [http://www.cs.rice.edu/~tlogic/Database/all-lectures.pdf Logic and Database Queries (readable)]
* [http://magenta.ruc.dk/dat_en/research/reports/127.pdf logic in databases workshop (complex stuff)]

See also about RDF and related:

* [http://cs.ttu.ee/kursused/wav4130/rdftutapp.pdf rdf tutorial addendum for logic and dbases]
* [http://en.wikipedia.org/wiki/Resource_Description_Framework RDF wikipedia]
* [http://www.w3.org/TR/rdf-primer/ w3c RDF primer]
* [http://en.wikipedia.org/wiki/RDFa wiki intro]
* [[Media:portaalidekoosvoime.ppt]] or as pdf [[Media:portaalidekoosvoime.pdf]]
* [http://www.w3.org/2006/07/SWD/RDFa/primer/ w3c rdfa primer]
* [http://tripletalk.wordpress.com/2011/01/25/rdfa-deployment-across-the-web/ rdfa and microformats deployment now]
* [http://www.microformats.org microformats: a similar technology]
* [http://www.w3.org/TR/2003/NOTE-lbase-20031010/ Hayes & Guha Lbase]

=== Rules in logic: apps in planning, robotics, ... ===

Lecture material:
* [[Media:Kr_rules1.pdf|Rules and logic: refresher]] and as [[Media:Kr_rules1.ppt|ppt]]

Topics:

* propositional, first order, higher order
* semantics/models
* proofs and deriving new information
* automated reasoning

See also:

* [[Media:Geoffreasoningnotes.pdf|Geoff reasoning course notes: just for background, no questions in exam]]
* [http://www.ee.pdx.edu/~mperkows/May4/005.Ch11-planning.ppt planning and blocks world]
* [[blocks world axiom/query examples from lecture]]

=== Various rule/logic languages ===

Lecture material:
* [[Media:Kr_rdfs.pdf|RDFS: rdf schema]] and as [[Media:Kr_rdfs.ppt|ppt]]
* [[Notes about RDF and OWL: logical meaning]]

Several languages:
* RDFS
* OWL
* KIF
* CL
* ontologies
* wordnet
* cyc
* restricted english systems

See also:
* [http://en.wikipedia.org/wiki/Resource_Description_Framework RDF wikipedia]
* [http://www.w3.org/TR/rdf-primer/ w3c RDF primer]
* [http://en.wikipedia.org/wiki/Web_Ontology_Language OWL wikipedia]
* [http://www.w3.org/TR/owl-guide/ w3c OWL guide]
* [http://cl.tamu.edu/ CL main page],
* [http://attempto.ifi.unizh.ch/site/pubs/papers/phd_kaljurand.pdf Attempto restricted english]

=== Scraping semistructured data ===

* Example for background: http://www.sightsmap.com/

=== Time, context, metainformation ===

=== Probabilistic and nonmonotonic reasoning ===

See also:
* [http://plato.stanford.edu/entries/logic-nonmonotonic/ nonmonotonic logic]
* [http://en.wikipedia.org/wiki/Default_logic default logic]

=== Personalized search and recommendations ===

=== Fast search of facts and rules using indexing ===

* [http://www.lambda.ee/index.php/Pilt:IndexNotes04.ppt G. Molina lecture 4]
* [http://www.lambda.ee/index.php/Pilt:IndexNotes05.ppt G. Molina lecture 5]
* [http://www.lambda.ee/index.php/Pilt:Fulltext_indexes.ppt good intro and overview]
* [http://www.lambda.ee/index.php/Pilt:Mccuneindexing.pdf mccune paper about term indexes]

=== Learning ===

See also:

* [http://www.cs.toronto.edu/~hinton/csc2515/notes/lec1.ppt intro]
* [http://gamma.cs.unc.edu/rob-course/public_html/LEC/MachineLearningMethods.ppt robots/cars]
* [http://www.cs.toronto.edu/~hinton/csc2515/notes/lec4.ppt backpropagation]
* [http://www.cs.toronto.edu/~hinton/csc2515/lectures.html good set of lecture materials]
* [http://www.cs.cmu.edu/~tom/mlbook-chapter-slides.html another good full set of lecture materials]
** sh [http://www.cs.cmu.edu/afs/cs.cmu.edu/project/theo-20/www/mlbook/ch4.pdf neural netw chapter]





[[Category:Kursus]]

Teadmiste otsing, formaliseerimine ja hoidmine

2012-02-20T16:11:39Z

Kalle: /* First lab */

<table>
<tr>
<td>
[[Pilt:Mephistophiles small.gif| left|teadmised]]
</td>
<td>
Code: ITV0060 
Link: http://www.lambda.ee/index.php/Teadmiste_otsing,_formaliseerimine_ja_hoidmine or http://www.lambda.ee/index/itv0060 
Lecturer: Tanel Tammet 
Contact: tammet@staff.ttu.ee, 6203457, TTÜ AK223 
Archives of previous years: [[itv0060 2011 arhiiv|2011]], [[itv0060 20010 arhiiv|2010]], [[itv0060 2009 arhiiv|2009]], [[itv0060 2008 arhiiv|2008]], [[itv0060 2007 arhiiv|2007]], [[itv0060 2006 arhiiv|2006]], [http:/cs.ttu.ee/kursused/wav4130/ older].
</td>
</tr>
</table>

== Time, place, result ==

Semester: spring 
Grading: exam 
Points: 3.5 

Lectures: every Monday 12.00-13.30, room IT-137a 
Practical work: Mondays on odd weeks 10.00-11.30, room IT-213H: 31 jan, 14 feb, 28 feb, 14 mar, 28 mar, 4 apr, 18 apr, 2 may, 16 may.

Practical work will give 40% and exam 60% of points underlying the final grade.

== Järeleksam 2011 sügis ==

Järeleksam toimub 26 august (reede): kell 10 auditoorium IT-137a

== Eksam ja konsultatsioon ==

16. mai: eksamiks õppimise materjalide ülevaade ja fikseerimine.

Eksam:

järeleksam 15 juuni (kolmapäev): kell 9 auditoorium IT-137a

15 juuni tulemused:, eksamipunktid viidud 0-60 skaalale:
* HP eksam 52p + 40p ettekanded, hinne 5
* GJ eksam 31p + 32p ettekanded, hinne 2

3 juuni (reede): kell 9 auditoorium VI-121

3 juuni tulemused:, eksamipunktid viidud 0-60 skaalale:
* TT eksam 36p + 40p ettekanded, hinne 3
* RL eksam 40p + 40p ettekanded, hinne 4
* HP eksam 28p + 40p ettekanded, hinne 2
* VP eksam 22p + 40p ettekanded, hinne 2
* DA eksam 57p + 40p ettekanded, hinne 5

8 juuni (kolmapäev): kell 9 auditoorium VII-131

8 juuni tulemused:, eksamipunktid viidud 0-60 skaalale:
* KL eksam 60p + 40p ettekanded, hinne 5
* ML eksam 51p + 40p ettekanded, hinne 5
* ET eksam 48p + 30p ettekanded, hinne 3
* DF eksam 58p + 40p ettekanded, hinne 5

Järeleksam kevadel:

* 15 juuni (kolmapäev): kell 9 auditoorium IT-137a

== Practical work ==

There are two labs on the same theme and an optional third lab.

=== First lab ===

The main goal of the first lab is to web-scrape data from the public sources using search engines and simple text analysis/statistics.

Details in [[Teadmised first lab 2012]]. The lab should be completed by March, later presentation will cut the points to half.

=== Second lab ===

The main goal of the second lab is to augment data obtained from the first lab by writing and using probabilistic rules with a rule engine.

Details in [[Teadmised second lab 2011]].

=== Optional third lab ===

The optional third lab can be done in parallel with the second. It will be a relatively complex lab: successful passing will give additional 20 points, thus the weight of the exam is diminished from 60% to 40% of the overall score.

What to do: use Attempto to create a system such that you can write rules and/or facts and/or queries for the second lab in restricted english. These restricted english sentences have to be automatically converted to the form usable by Otter. It is a good idea to avoid probabilities in these rules, since it would be very hard to encode probabilities using attempto.

See: http://attempto.ifi.uzh.ch/site/ , http://attempto.ifi.uzh.ch/ape/

== Materials to study for the exam ==

Eksamiküsimused:

Väikesed ülesanded: data ja reeglite
esitamine ja kasutamine

Andmebaas nö tüüpilise relatsioonilise
mudelina, key-value ja rdf kujul

Ülesanne blocks worldist: esita mingi
olukord, mingi reegel ja näita rakendamise
tulemust

RDF, RDFS ja OWL põhiasjad: väike
ülesanne. Süntaksid ja eridetailid
ei ole olulised.

RDF ja RDFS esitada loogikas
faktide ja reeglitena, mõni
lihtne OWLi näide võib ka olla

Arusaamine RDFa põhiasjadest, oskus
teha näidet.

MÕni üldküsimus Nelli ja Attempto
kohta.

Arusaam et mis on mittemonotoonne
loogika ja mis ei ole, mis oleks
viisid esitada vasturääkivaid
teadmisi. Võib olla väike ülesanne
mingite asjade esitamise kohta: näide
ja selgitus.

Default loogika ülesanne a la linnud
ja Quaker diamond: panna kirja reeglid
ja viia läbi järeldused ja vbl selgitada
lisaks. Seos CWA, Prolog ja default loogika
vahel võib samuti küsimuse alla tulla.

Indeksid: arusaam baasindekseerimis-ideedest
ja B+ treest. Ntx ehitada B+ mingi
väikese datahulga jaoks. Sama hashi kohta
ja bit indeksite kohta. Eriteemad a la
indeksi pakkimine ja erinevad hashi
viisid ei tule teemaks.

Täistekstiindeksid: osata ehitada näite
pealt. Vt presentatsioonist erinevaid
viise, ntx Patricia trie.

Termiindeksid: piisab discrimination
tree indeksist ja näite põhjal
indeksi ehitamisest.

=== Declarative and procedural representations ===

Lecture material:
* [[Media:Kr intro.ppt | Intro lecture: declarative and procedural representations]]

=== Databases: meaning and representation of facts ===

* [[Media:Kr_plain_data_db_logic.ppt | relation of plain data in databases to logic & representing complex structures in databases ]]
* [[Media:Kr_rdf_schemalessdb_3.pdf|key/value pairs and rdf, compared to the relational model]]

=== Rules in logic: apps in planning, robotics, ... ===

Lecture material:
* [[Media:Kr_rules1.pdf|Rules and logic: refresher]] and as [[Media:Kr_rules1.ppt|ppt]]

See also:

* [[Media:Geoffreasoningnotes.pdf|Geoff reasoning course notes: just for background, no questions in exam]]
* [http://www.ee.pdx.edu/~mperkows/May4/005.Ch11-planning.ppt planning and blocks world]
* [[blocks world axiom/query examples from lecture]]

=== RDF, RDFS, OWL and conversion to 1st order logic ===

Lecture material:
* [[Media:Kr_rdfs.pdf|RDFS: rdf schema]] and as [[Media:Kr_rdfs.ppt|ppt]]

Understand RDF and RDFS:

* [http://en.wikipedia.org/wiki/Resource_Description_Framework RDF wikipedia]
* [http://www.w3.org/TR/rdf-primer/ w3c RDF primer]

Understand basics of owl:

* [http://en.wikipedia.org/wiki/Web_Ontology_Language OWL wikipedia]
* [http://www.w3.org/TR/owl-guide/ w3c OWL guide]

=== RDFa, reading and using RDF data from web pages ===

Understand main parts:

* [http://en.wikipedia.org/wiki/RDFa wiki intro]
* [[Media:portaalidekoosvoime.ppt]] or as pdf [[Media:portaalidekoosvoime.pdf]]
* [http://www.w3.org/2006/07/SWD/RDFa/primer/ w3c rdfa primer]

=== RDF relation to logic ===

Notes from lecture:
* [[Notes about RDF and OWL: logical meaning]]

Understand a bit of:
* [http://www.w3.org/TR/2003/NOTE-lbase-20031010/ Hayes & Guha Lbase]

=== Semantics of ordinary relational base ===

* [http://cs.ttu.ee/kursused/wav4130/rdftutapp.pdf rdf tutorial addendum for logic and dbases]

=== First order logic in query engines and planning ===

* [[Media:Geoffreasoningnotes.pdf|Geoff reasoning course notes: just for background, no questions in exam]]
* [http://www.ee.pdx.edu/~mperkows/May4/005.Ch11-planning.ppt planning and blocks world]
* [[blocks world axiom/query examples from lecture]]

=== Data extraction ===

Web mining by Nell:
* [http://rtw.ml.cmu.edu/rtw/ Nell]

=== Restricted english jms ===

* [http://attempto.ifi.unizh.ch/site/pubs/papers/phd_kaljurand.pdf Attempto restricted english]

=== Uncertain knowledge logic ===

* [http://plato.stanford.edu/entries/logic-nonmonotonic/ nonmonotonic logic]
* [http://en.wikipedia.org/wiki/Default_logic default logic]
* [http://sc.snu.ac.kr/courses/2008/spring/pg/krr/Lectures/Lecture11/Lec11.ppt Reasoning with Conflicting Knowledge]

=== Indexes ===

Traditional database indexes incl B+ tree:
* [http://www.lambda.ee/index.php/Pilt:IndexNotes04.ppt G. Molina lecture 4]

Bitmap indexes:
* [http://en.wikipedia.org/wiki/Bitmap_index wiki]

Hash indexes:
* [http://en.wikipedia.org/wiki/Hash_table wiki]

Fulltext indexes:
* [http://www.lambda.ee/index.php/Pilt:Fulltext_indexes.ppt good intro and overview]

Term indexes:
Understand basics of discrimination tree trie-type term indexes and path indexes from
* [http://www.lambda.ee/index.php/Pilt:Mccuneindexing.pdf mccune paper]

== Course structure ==

We start with representing and converting facts, continue to rules, first monotonic, then probabilistic and nonmonotonic, then look into search and indexing of complex information, and finally, some learning.

=== Intro and paradigms of knowledge representation ===

Lecture materials:
* [[Media:Kr intro.ppt | Intro lecture: declarative and procedural representations]]
* [http://dl.dropbox.com/u/11794296/roboswarm_saso_2.pdf roboswarm example of combining dec and proc]

For KR, declarative and procedural rep see:
* [http://en.wikipedia.org/wiki/Knowledge_representation Short wiki overview of knowledge representation]
* [http://groups.csail.mit.edu/medg/ftp/psz/k-rep.html Long and philosophical MIT overview of knowledge representation]
* [http://www.cse.buffalo.edu/~rapaport/676/F01/proc.decl.html Pointers to classical papers]
* [http://www.cogarch.org/index.php/Main_Page A Survey of Cognitive and Agent Architectures]
* [ftp://reports.stanford.edu/pub/cstr/reports/cs/tr/76/581/CS-TR-76-581.pdf Classical KRL paper from 1977: Bobrow & Winograd]
Web mining by Nell:
* [http://rtw.ml.cmu.edu/rtw/ Nell]
* [http://www.cs.cmu.edu/~rcwang/ Richard Wang]
* [http://www.boowa.com/ boowa extractor]

=== Databases: meaning and representation of facts ===

Lecture material:


* [[Media:Kr_plain_data_db_logic.ppt | relation of plain data in databases to logic & representing complex structures in databases ]]

* [[Media:Kr_rdf_schemalessdb_3.pdf|key/value pairs and rdf, compared to the relational model]]

See also about databases and logic:

* [http://www.cs.rice.edu/~tlogic/Database/all-lectures.pdf Logic and Database Queries (readable)]
* [http://magenta.ruc.dk/dat_en/research/reports/127.pdf logic in databases workshop (complex stuff)]

See also about RDF and related:

* [http://cs.ttu.ee/kursused/wav4130/rdftutapp.pdf rdf tutorial addendum for logic and dbases]
* [http://en.wikipedia.org/wiki/Resource_Description_Framework RDF wikipedia]
* [http://www.w3.org/TR/rdf-primer/ w3c RDF primer]
* [http://en.wikipedia.org/wiki/RDFa wiki intro]
* [[Media:portaalidekoosvoime.ppt]] or as pdf [[Media:portaalidekoosvoime.pdf]]
* [http://www.w3.org/2006/07/SWD/RDFa/primer/ w3c rdfa primer]
* [http://tripletalk.wordpress.com/2011/01/25/rdfa-deployment-across-the-web/ rdfa and microformats deployment now]
* [http://www.microformats.org microformats: a similar technology]
* [http://www.w3.org/TR/2003/NOTE-lbase-20031010/ Hayes & Guha Lbase]

=== Rules in logic: apps in planning, robotics, ... ===

Lecture material:
* [[Media:Kr_rules1.pdf|Rules and logic: refresher]] and as [[Media:Kr_rules1.ppt|ppt]]

Topics:

* propositional, first order, higher order
* semantics/models
* proofs and deriving new information
* automated reasoning

See also:

* [[Media:Geoffreasoningnotes.pdf|Geoff reasoning course notes: just for background, no questions in exam]]
* [http://www.ee.pdx.edu/~mperkows/May4/005.Ch11-planning.ppt planning and blocks world]
* [[blocks world axiom/query examples from lecture]]

=== Various rule/logic languages ===

Lecture material:
* [[Media:Kr_rdfs.pdf|RDFS: rdf schema]] and as [[Media:Kr_rdfs.ppt|ppt]]
* [[Notes about RDF and OWL: logical meaning]]

Several languages:
* RDFS
* OWL
* KIF
* CL
* ontologies
* wordnet
* cyc
* restricted english systems

See also:
* [http://en.wikipedia.org/wiki/Resource_Description_Framework RDF wikipedia]
* [http://www.w3.org/TR/rdf-primer/ w3c RDF primer]
* [http://en.wikipedia.org/wiki/Web_Ontology_Language OWL wikipedia]
* [http://www.w3.org/TR/owl-guide/ w3c OWL guide]
* [http://cl.tamu.edu/ CL main page],
* [http://attempto.ifi.unizh.ch/site/pubs/papers/phd_kaljurand.pdf Attempto restricted english]

=== Scraping semistructured data ===

* Example for background: http://www.sightsmap.com/

=== Time, context, metainformation ===

=== Probabilistic and nonmonotonic reasoning ===

See also:
* [http://plato.stanford.edu/entries/logic-nonmonotonic/ nonmonotonic logic]
* [http://en.wikipedia.org/wiki/Default_logic default logic]

=== Personalized search and recommendations ===

=== Fast search of facts and rules using indexing ===

* [http://www.lambda.ee/index.php/Pilt:IndexNotes04.ppt G. Molina lecture 4]
* [http://www.lambda.ee/index.php/Pilt:IndexNotes05.ppt G. Molina lecture 5]
* [http://www.lambda.ee/index.php/Pilt:Fulltext_indexes.ppt good intro and overview]
* [http://www.lambda.ee/index.php/Pilt:Mccuneindexing.pdf mccune paper about term indexes]

=== Learning ===

See also:

* [http://www.cs.toronto.edu/~hinton/csc2515/notes/lec1.ppt intro]
* [http://gamma.cs.unc.edu/rob-course/public_html/LEC/MachineLearningMethods.ppt robots/cars]
* [http://www.cs.toronto.edu/~hinton/csc2515/notes/lec4.ppt backpropagation]
* [http://www.cs.toronto.edu/~hinton/csc2515/lectures.html good set of lecture materials]
* [http://www.cs.cmu.edu/~tom/mlbook-chapter-slides.html another good full set of lecture materials]
** sh [http://www.cs.cmu.edu/afs/cs.cmu.edu/project/theo-20/www/mlbook/ch4.pdf neural netw chapter]





[[Category:Kursus]]

Itv0060 2011 arhiiv

2012-02-20T16:06:16Z

Kalle:

<table>
<tr>
<td>
[[Pilt:Mephistophiles small.gif| left|teadmised]]
</td>
<td>
Code: ITV0060 
Link: http://www.lambda.ee/index.php/Teadmiste_otsing,_formaliseerimine_ja_hoidmine or http://www.lambda.ee/index/itv0060 
Lecturer: Tanel Tammet 
Contact: tammet@staff.ttu.ee, 6203457, TTÜ AK223 
Archives of previous years: [[itv0060 20010 arhiiv|2010]], [[itv0060 2009 arhiiv|2009]], [[itv0060 2008 arhiiv|2008]], [[itv0060 2007 arhiiv|2007]], [[itv0060 2006 arhiiv|2006]], [http:/cs.ttu.ee/kursused/wav4130/ older].
</td>
</tr>
</table>

== Time, place, result ==

Semester: spring 
Grading: exam 
Points: 3.5 

Lectures: every Monday 12.00-13.30, room IT-137a 
Practical work: Mondays on odd weeks 10.00-11.30, room IT-213H: 31 jan, 14 feb, 28 feb, 14 mar, 28 mar, 4 apr, 18 apr, 2 may, 16 may.

Practical work will give 40% and exam 60% of points underlying the final grade.

== Järeleksam 2011 sügis ==

Järeleksam toimub 26 august (reede): kell 10 auditoorium IT-137a

== Eksam ja konsultatsioon ==

16. mai: eksamiks õppimise materjalide ülevaade ja fikseerimine.

Eksam:

järeleksam 15 juuni (kolmapäev): kell 9 auditoorium IT-137a

15 juuni tulemused:, eksamipunktid viidud 0-60 skaalale:
* HP eksam 52p + 40p ettekanded, hinne 5
* GJ eksam 31p + 32p ettekanded, hinne 2

3 juuni (reede): kell 9 auditoorium VI-121

3 juuni tulemused:, eksamipunktid viidud 0-60 skaalale:
* TT eksam 36p + 40p ettekanded, hinne 3
* RL eksam 40p + 40p ettekanded, hinne 4
* HP eksam 28p + 40p ettekanded, hinne 2
* VP eksam 22p + 40p ettekanded, hinne 2
* DA eksam 57p + 40p ettekanded, hinne 5

8 juuni (kolmapäev): kell 9 auditoorium VII-131

8 juuni tulemused:, eksamipunktid viidud 0-60 skaalale:
* KL eksam 60p + 40p ettekanded, hinne 5
* ML eksam 51p + 40p ettekanded, hinne 5
* ET eksam 48p + 30p ettekanded, hinne 3
* DF eksam 58p + 40p ettekanded, hinne 5

Järeleksam kevadel:

* 15 juuni (kolmapäev): kell 9 auditoorium IT-137a

== Practical work ==

There are two labs on the same theme and an optional third lab.

=== First lab ===

The main goal of the first lab is to web-scrape data from the public sources using search engines and simple text analysis/statistics.

Details in [[Teadmised first lab 2011]]. The lab should be completed by March, later presentation will cut the points to half.

=== Second lab ===

The main goal of the second lab is to augment data obtained from the first lab by writing and using probabilistic rules with a rule engine.

Details in [[Teadmised second lab 2011]].

=== Optional third lab ===

The optional third lab can be done in parallel with the second. It will be a relatively complex lab: successful passing will give additional 20 points, thus the weight of the exam is diminished from 60% to 40% of the overall score.

What to do: use Attempto to create a system such that you can write rules and/or facts and/or queries for the second lab in restricted english. These restricted english sentences have to be automatically converted to the form usable by Otter. It is a good idea to avoid probabilities in these rules, since it would be very hard to encode probabilities using attempto.

See: http://attempto.ifi.uzh.ch/site/ , http://attempto.ifi.uzh.ch/ape/

== Materials to study for the exam ==

Eksamiküsimused:

Väikesed ülesanded: data ja reeglite
esitamine ja kasutamine

Andmebaas nö tüüpilise relatsioonilise
mudelina, key-value ja rdf kujul

Ülesanne blocks worldist: esita mingi
olukord, mingi reegel ja näita rakendamise
tulemust

RDF, RDFS ja OWL põhiasjad: väike
ülesanne. Süntaksid ja eridetailid
ei ole olulised.

RDF ja RDFS esitada loogikas
faktide ja reeglitena, mõni
lihtne OWLi näide võib ka olla

Arusaamine RDFa põhiasjadest, oskus
teha näidet.

MÕni üldküsimus Nelli ja Attempto
kohta.

Arusaam et mis on mittemonotoonne
loogika ja mis ei ole, mis oleks
viisid esitada vasturääkivaid
teadmisi. Võib olla väike ülesanne
mingite asjade esitamise kohta: näide
ja selgitus.

Default loogika ülesanne a la linnud
ja Quaker diamond: panna kirja reeglid
ja viia läbi järeldused ja vbl selgitada
lisaks. Seos CWA, Prolog ja default loogika
vahel võib samuti küsimuse alla tulla.

Indeksid: arusaam baasindekseerimis-ideedest
ja B+ treest. Ntx ehitada B+ mingi
väikese datahulga jaoks. Sama hashi kohta
ja bit indeksite kohta. Eriteemad a la
indeksi pakkimine ja erinevad hashi
viisid ei tule teemaks.

Täistekstiindeksid: osata ehitada näite
pealt. Vt presentatsioonist erinevaid
viise, ntx Patricia trie.

Termiindeksid: piisab discrimination
tree indeksist ja näite põhjal
indeksi ehitamisest.

=== Declarative and procedural representations ===

Lecture material:
* [[Media:Kr intro.ppt | Intro lecture: declarative and procedural representations]]

=== Databases: meaning and representation of facts ===

* [[Media:Kr_plain_data_db_logic.ppt | relation of plain data in databases to logic & representing complex structures in databases ]]
* [[Media:Kr_rdf_schemalessdb_3.pdf|key/value pairs and rdf, compared to the relational model]]

=== Rules in logic: apps in planning, robotics, ... ===

Lecture material:
* [[Media:Kr_rules1.pdf|Rules and logic: refresher]] and as [[Media:Kr_rules1.ppt|ppt]]

See also:

* [[Media:Geoffreasoningnotes.pdf|Geoff reasoning course notes: just for background, no questions in exam]]
* [http://www.ee.pdx.edu/~mperkows/May4/005.Ch11-planning.ppt planning and blocks world]
* [[blocks world axiom/query examples from lecture]]

=== RDF, RDFS, OWL and conversion to 1st order logic ===

Lecture material:
* [[Media:Kr_rdfs.pdf|RDFS: rdf schema]] and as [[Media:Kr_rdfs.ppt|ppt]]

Understand RDF and RDFS:

* [http://en.wikipedia.org/wiki/Resource_Description_Framework RDF wikipedia]
* [http://www.w3.org/TR/rdf-primer/ w3c RDF primer]

Understand basics of owl:

* [http://en.wikipedia.org/wiki/Web_Ontology_Language OWL wikipedia]
* [http://www.w3.org/TR/owl-guide/ w3c OWL guide]

=== RDFa, reading and using RDF data from web pages ===

Understand main parts:

* [http://en.wikipedia.org/wiki/RDFa wiki intro]
* [[Media:portaalidekoosvoime.ppt]] or as pdf [[Media:portaalidekoosvoime.pdf]]
* [http://www.w3.org/2006/07/SWD/RDFa/primer/ w3c rdfa primer]

=== RDF relation to logic ===

Notes from lecture:
* [[Notes about RDF and OWL: logical meaning]]

Understand a bit of:
* [http://www.w3.org/TR/2003/NOTE-lbase-20031010/ Hayes & Guha Lbase]

=== Semantics of ordinary relational base ===

* [http://cs.ttu.ee/kursused/wav4130/rdftutapp.pdf rdf tutorial addendum for logic and dbases]

=== First order logic in query engines and planning ===

* [[Media:Geoffreasoningnotes.pdf|Geoff reasoning course notes: just for background, no questions in exam]]
* [http://www.ee.pdx.edu/~mperkows/May4/005.Ch11-planning.ppt planning and blocks world]
* [[blocks world axiom/query examples from lecture]]

=== Data extraction ===

Web mining by Nell:
* [http://rtw.ml.cmu.edu/rtw/ Nell]

=== Restricted english jms ===

* [http://attempto.ifi.unizh.ch/site/pubs/papers/phd_kaljurand.pdf Attempto restricted english]

=== Uncertain knowledge logic ===

* [http://plato.stanford.edu/entries/logic-nonmonotonic/ nonmonotonic logic]
* [http://en.wikipedia.org/wiki/Default_logic default logic]
* [http://sc.snu.ac.kr/courses/2008/spring/pg/krr/Lectures/Lecture11/Lec11.ppt Reasoning with Conflicting Knowledge]

=== Indexes ===

Traditional database indexes incl B+ tree:
* [http://www.lambda.ee/index.php/Pilt:IndexNotes04.ppt G. Molina lecture 4]

Bitmap indexes:
* [http://en.wikipedia.org/wiki/Bitmap_index wiki]

Hash indexes:
* [http://en.wikipedia.org/wiki/Hash_table wiki]

Fulltext indexes:
* [http://www.lambda.ee/index.php/Pilt:Fulltext_indexes.ppt good intro and overview]

Term indexes:
Understand basics of discrimination tree trie-type term indexes and path indexes from
* [http://www.lambda.ee/index.php/Pilt:Mccuneindexing.pdf mccune paper]

== Course structure ==

We start with representing and converting facts, continue to rules, first monotonic, then probabilistic and nonmonotonic, then look into search and indexing of complex information, and finally, some learning.

=== Intro and paradigms of knowledge representation ===

Lecture materials:
* [[Media:Kr intro.ppt | Intro lecture: declarative and procedural representations]]
* [http://dl.dropbox.com/u/11794296/roboswarm_saso_2.pdf roboswarm example of combining dec and proc]

For KR, declarative and procedural rep see:
* [http://en.wikipedia.org/wiki/Knowledge_representation Short wiki overview of knowledge representation]
* [http://groups.csail.mit.edu/medg/ftp/psz/k-rep.html Long and philosophical MIT overview of knowledge representation]
* [http://www.cse.buffalo.edu/~rapaport/676/F01/proc.decl.html Pointers to classical papers]
* [http://www.cogarch.org/index.php/Main_Page A Survey of Cognitive and Agent Architectures]
* [ftp://reports.stanford.edu/pub/cstr/reports/cs/tr/76/581/CS-TR-76-581.pdf Classical KRL paper from 1977: Bobrow & Winograd]
Web mining by Nell:
* [http://rtw.ml.cmu.edu/rtw/ Nell]
* [http://www.cs.cmu.edu/~rcwang/ Richard Wang]
* [http://www.boowa.com/ boowa extractor]

=== Databases: meaning and representation of facts ===

Lecture material:


* [[Media:Kr_plain_data_db_logic.ppt | relation of plain data in databases to logic & representing complex structures in databases ]]

* [[Media:Kr_rdf_schemalessdb_3.pdf|key/value pairs and rdf, compared to the relational model]]

See also about databases and logic:

* [http://www.cs.rice.edu/~tlogic/Database/all-lectures.pdf Logic and Database Queries (readable)]
* [http://magenta.ruc.dk/dat_en/research/reports/127.pdf logic in databases workshop (complex stuff)]

See also about RDF and related:

* [http://cs.ttu.ee/kursused/wav4130/rdftutapp.pdf rdf tutorial addendum for logic and dbases]
* [http://en.wikipedia.org/wiki/Resource_Description_Framework RDF wikipedia]
* [http://www.w3.org/TR/rdf-primer/ w3c RDF primer]
* [http://en.wikipedia.org/wiki/RDFa wiki intro]
* [[Media:portaalidekoosvoime.ppt]] or as pdf [[Media:portaalidekoosvoime.pdf]]
* [http://www.w3.org/2006/07/SWD/RDFa/primer/ w3c rdfa primer]
* [http://tripletalk.wordpress.com/2011/01/25/rdfa-deployment-across-the-web/ rdfa and microformats deployment now]
* [http://www.microformats.org microformats: a similar technology]
* [http://www.w3.org/TR/2003/NOTE-lbase-20031010/ Hayes & Guha Lbase]

=== Rules in logic: apps in planning, robotics, ... ===

Lecture material:
* [[Media:Kr_rules1.pdf|Rules and logic: refresher]] and as [[Media:Kr_rules1.ppt|ppt]]

Topics:

* propositional, first order, higher order
* semantics/models
* proofs and deriving new information
* automated reasoning

See also:

* [[Media:Geoffreasoningnotes.pdf|Geoff reasoning course notes: just for background, no questions in exam]]
* [http://www.ee.pdx.edu/~mperkows/May4/005.Ch11-planning.ppt planning and blocks world]
* [[blocks world axiom/query examples from lecture]]

=== Various rule/logic languages ===

Lecture material:
* [[Media:Kr_rdfs.pdf|RDFS: rdf schema]] and as [[Media:Kr_rdfs.ppt|ppt]]
* [[Notes about RDF and OWL: logical meaning]]

Several languages:
* RDFS
* OWL
* KIF
* CL
* ontologies
* wordnet
* cyc
* restricted english systems

See also:
* [http://en.wikipedia.org/wiki/Resource_Description_Framework RDF wikipedia]
* [http://www.w3.org/TR/rdf-primer/ w3c RDF primer]
* [http://en.wikipedia.org/wiki/Web_Ontology_Language OWL wikipedia]
* [http://www.w3.org/TR/owl-guide/ w3c OWL guide]
* [http://cl.tamu.edu/ CL main page],
* [http://attempto.ifi.unizh.ch/site/pubs/papers/phd_kaljurand.pdf Attempto restricted english]

=== Scraping semistructured data ===

* Example for background: http://www.sightsmap.com/

=== Time, context, metainformation ===

=== Probabilistic and nonmonotonic reasoning ===

See also:
* [http://plato.stanford.edu/entries/logic-nonmonotonic/ nonmonotonic logic]
* [http://en.wikipedia.org/wiki/Default_logic default logic]

=== Personalized search and recommendations ===

=== Fast search of facts and rules using indexing ===

* [http://www.lambda.ee/index.php/Pilt:IndexNotes04.ppt G. Molina lecture 4]
* [http://www.lambda.ee/index.php/Pilt:IndexNotes05.ppt G. Molina lecture 5]
* [http://www.lambda.ee/index.php/Pilt:Fulltext_indexes.ppt good intro and overview]
* [http://www.lambda.ee/index.php/Pilt:Mccuneindexing.pdf mccune paper about term indexes]

=== Learning ===

See also:

* [http://www.cs.toronto.edu/~hinton/csc2515/notes/lec1.ppt intro]
* [http://gamma.cs.unc.edu/rob-course/public_html/LEC/MachineLearningMethods.ppt robots/cars]
* [http://www.cs.toronto.edu/~hinton/csc2515/notes/lec4.ppt backpropagation]
* [http://www.cs.toronto.edu/~hinton/csc2515/lectures.html good set of lecture materials]
* [http://www.cs.cmu.edu/~tom/mlbook-chapter-slides.html another good full set of lecture materials]
** sh [http://www.cs.cmu.edu/afs/cs.cmu.edu/project/theo-20/www/mlbook/ch4.pdf neural netw chapter]





[[Category:Kursus]]

Teadmiste otsing, formaliseerimine ja hoidmine

2012-02-20T16:06:02Z

Kalle:

<table>
<tr>
<td>
[[Pilt:Mephistophiles small.gif| left|teadmised]]
</td>
<td>
Code: ITV0060 
Link: http://www.lambda.ee/index.php/Teadmiste_otsing,_formaliseerimine_ja_hoidmine or http://www.lambda.ee/index/itv0060 
Lecturer: Tanel Tammet 
Contact: tammet@staff.ttu.ee, 6203457, TTÜ AK223 
Archives of previous years: [[itv0060 2011 arhiiv|2011]], [[itv0060 20010 arhiiv|2010]], [[itv0060 2009 arhiiv|2009]], [[itv0060 2008 arhiiv|2008]], [[itv0060 2007 arhiiv|2007]], [[itv0060 2006 arhiiv|2006]], [http:/cs.ttu.ee/kursused/wav4130/ older].
</td>
</tr>
</table>

== Time, place, result ==

Semester: spring 
Grading: exam 
Points: 3.5 

Lectures: every Monday 12.00-13.30, room IT-137a 
Practical work: Mondays on odd weeks 10.00-11.30, room IT-213H: 31 jan, 14 feb, 28 feb, 14 mar, 28 mar, 4 apr, 18 apr, 2 may, 16 may.

Practical work will give 40% and exam 60% of points underlying the final grade.

== Järeleksam 2011 sügis ==

Järeleksam toimub 26 august (reede): kell 10 auditoorium IT-137a

== Eksam ja konsultatsioon ==

16. mai: eksamiks õppimise materjalide ülevaade ja fikseerimine.

Eksam:

järeleksam 15 juuni (kolmapäev): kell 9 auditoorium IT-137a

15 juuni tulemused:, eksamipunktid viidud 0-60 skaalale:
* HP eksam 52p + 40p ettekanded, hinne 5
* GJ eksam 31p + 32p ettekanded, hinne 2

3 juuni (reede): kell 9 auditoorium VI-121

3 juuni tulemused:, eksamipunktid viidud 0-60 skaalale:
* TT eksam 36p + 40p ettekanded, hinne 3
* RL eksam 40p + 40p ettekanded, hinne 4
* HP eksam 28p + 40p ettekanded, hinne 2
* VP eksam 22p + 40p ettekanded, hinne 2
* DA eksam 57p + 40p ettekanded, hinne 5

8 juuni (kolmapäev): kell 9 auditoorium VII-131

8 juuni tulemused:, eksamipunktid viidud 0-60 skaalale:
* KL eksam 60p + 40p ettekanded, hinne 5
* ML eksam 51p + 40p ettekanded, hinne 5
* ET eksam 48p + 30p ettekanded, hinne 3
* DF eksam 58p + 40p ettekanded, hinne 5

Järeleksam kevadel:

* 15 juuni (kolmapäev): kell 9 auditoorium IT-137a

== Practical work ==

There are two labs on the same theme and an optional third lab.

=== First lab ===

The main goal of the first lab is to web-scrape data from the public sources using search engines and simple text analysis/statistics.

Details in [[Teadmised first lab 2011]]. The lab should be completed by March, later presentation will cut the points to half.

=== Second lab ===

The main goal of the second lab is to augment data obtained from the first lab by writing and using probabilistic rules with a rule engine.

Details in [[Teadmised second lab 2011]].

=== Optional third lab ===

The optional third lab can be done in parallel with the second. It will be a relatively complex lab: successful passing will give additional 20 points, thus the weight of the exam is diminished from 60% to 40% of the overall score.

What to do: use Attempto to create a system such that you can write rules and/or facts and/or queries for the second lab in restricted english. These restricted english sentences have to be automatically converted to the form usable by Otter. It is a good idea to avoid probabilities in these rules, since it would be very hard to encode probabilities using attempto.

See: http://attempto.ifi.uzh.ch/site/ , http://attempto.ifi.uzh.ch/ape/

== Materials to study for the exam ==

Eksamiküsimused:

Väikesed ülesanded: data ja reeglite
esitamine ja kasutamine

Andmebaas nö tüüpilise relatsioonilise
mudelina, key-value ja rdf kujul

Ülesanne blocks worldist: esita mingi
olukord, mingi reegel ja näita rakendamise
tulemust

RDF, RDFS ja OWL põhiasjad: väike
ülesanne. Süntaksid ja eridetailid
ei ole olulised.

RDF ja RDFS esitada loogikas
faktide ja reeglitena, mõni
lihtne OWLi näide võib ka olla

Arusaamine RDFa põhiasjadest, oskus
teha näidet.

MÕni üldküsimus Nelli ja Attempto
kohta.

Arusaam et mis on mittemonotoonne
loogika ja mis ei ole, mis oleks
viisid esitada vasturääkivaid
teadmisi. Võib olla väike ülesanne
mingite asjade esitamise kohta: näide
ja selgitus.

Default loogika ülesanne a la linnud
ja Quaker diamond: panna kirja reeglid
ja viia läbi järeldused ja vbl selgitada
lisaks. Seos CWA, Prolog ja default loogika
vahel võib samuti küsimuse alla tulla.

Indeksid: arusaam baasindekseerimis-ideedest
ja B+ treest. Ntx ehitada B+ mingi
väikese datahulga jaoks. Sama hashi kohta
ja bit indeksite kohta. Eriteemad a la
indeksi pakkimine ja erinevad hashi
viisid ei tule teemaks.

Täistekstiindeksid: osata ehitada näite
pealt. Vt presentatsioonist erinevaid
viise, ntx Patricia trie.

Termiindeksid: piisab discrimination
tree indeksist ja näite põhjal
indeksi ehitamisest.

=== Declarative and procedural representations ===

Lecture material:
* [[Media:Kr intro.ppt | Intro lecture: declarative and procedural representations]]

=== Databases: meaning and representation of facts ===

* [[Media:Kr_plain_data_db_logic.ppt | relation of plain data in databases to logic & representing complex structures in databases ]]
* [[Media:Kr_rdf_schemalessdb_3.pdf|key/value pairs and rdf, compared to the relational model]]

=== Rules in logic: apps in planning, robotics, ... ===

Lecture material:
* [[Media:Kr_rules1.pdf|Rules and logic: refresher]] and as [[Media:Kr_rules1.ppt|ppt]]

See also:

* [[Media:Geoffreasoningnotes.pdf|Geoff reasoning course notes: just for background, no questions in exam]]
* [http://www.ee.pdx.edu/~mperkows/May4/005.Ch11-planning.ppt planning and blocks world]
* [[blocks world axiom/query examples from lecture]]

=== RDF, RDFS, OWL and conversion to 1st order logic ===

Lecture material:
* [[Media:Kr_rdfs.pdf|RDFS: rdf schema]] and as [[Media:Kr_rdfs.ppt|ppt]]

Understand RDF and RDFS:

* [http://en.wikipedia.org/wiki/Resource_Description_Framework RDF wikipedia]
* [http://www.w3.org/TR/rdf-primer/ w3c RDF primer]

Understand basics of owl:

* [http://en.wikipedia.org/wiki/Web_Ontology_Language OWL wikipedia]
* [http://www.w3.org/TR/owl-guide/ w3c OWL guide]

=== RDFa, reading and using RDF data from web pages ===

Understand main parts:

* [http://en.wikipedia.org/wiki/RDFa wiki intro]
* [[Media:portaalidekoosvoime.ppt]] or as pdf [[Media:portaalidekoosvoime.pdf]]
* [http://www.w3.org/2006/07/SWD/RDFa/primer/ w3c rdfa primer]

=== RDF relation to logic ===

Notes from lecture:
* [[Notes about RDF and OWL: logical meaning]]

Understand a bit of:
* [http://www.w3.org/TR/2003/NOTE-lbase-20031010/ Hayes & Guha Lbase]

=== Semantics of ordinary relational base ===

* [http://cs.ttu.ee/kursused/wav4130/rdftutapp.pdf rdf tutorial addendum for logic and dbases]

=== First order logic in query engines and planning ===

* [[Media:Geoffreasoningnotes.pdf|Geoff reasoning course notes: just for background, no questions in exam]]
* [http://www.ee.pdx.edu/~mperkows/May4/005.Ch11-planning.ppt planning and blocks world]
* [[blocks world axiom/query examples from lecture]]

=== Data extraction ===

Web mining by Nell:
* [http://rtw.ml.cmu.edu/rtw/ Nell]

=== Restricted english jms ===

* [http://attempto.ifi.unizh.ch/site/pubs/papers/phd_kaljurand.pdf Attempto restricted english]

=== Uncertain knowledge logic ===

* [http://plato.stanford.edu/entries/logic-nonmonotonic/ nonmonotonic logic]
* [http://en.wikipedia.org/wiki/Default_logic default logic]
* [http://sc.snu.ac.kr/courses/2008/spring/pg/krr/Lectures/Lecture11/Lec11.ppt Reasoning with Conflicting Knowledge]

=== Indexes ===

Traditional database indexes incl B+ tree:
* [http://www.lambda.ee/index.php/Pilt:IndexNotes04.ppt G. Molina lecture 4]

Bitmap indexes:
* [http://en.wikipedia.org/wiki/Bitmap_index wiki]

Hash indexes:
* [http://en.wikipedia.org/wiki/Hash_table wiki]

Fulltext indexes:
* [http://www.lambda.ee/index.php/Pilt:Fulltext_indexes.ppt good intro and overview]

Term indexes:
Understand basics of discrimination tree trie-type term indexes and path indexes from
* [http://www.lambda.ee/index.php/Pilt:Mccuneindexing.pdf mccune paper]

== Course structure ==

We start with representing and converting facts, continue to rules, first monotonic, then probabilistic and nonmonotonic, then look into search and indexing of complex information, and finally, some learning.

=== Intro and paradigms of knowledge representation ===

Lecture materials:
* [[Media:Kr intro.ppt | Intro lecture: declarative and procedural representations]]
* [http://dl.dropbox.com/u/11794296/roboswarm_saso_2.pdf roboswarm example of combining dec and proc]

For KR, declarative and procedural rep see:
* [http://en.wikipedia.org/wiki/Knowledge_representation Short wiki overview of knowledge representation]
* [http://groups.csail.mit.edu/medg/ftp/psz/k-rep.html Long and philosophical MIT overview of knowledge representation]
* [http://www.cse.buffalo.edu/~rapaport/676/F01/proc.decl.html Pointers to classical papers]
* [http://www.cogarch.org/index.php/Main_Page A Survey of Cognitive and Agent Architectures]
* [ftp://reports.stanford.edu/pub/cstr/reports/cs/tr/76/581/CS-TR-76-581.pdf Classical KRL paper from 1977: Bobrow & Winograd]
Web mining by Nell:
* [http://rtw.ml.cmu.edu/rtw/ Nell]
* [http://www.cs.cmu.edu/~rcwang/ Richard Wang]
* [http://www.boowa.com/ boowa extractor]

=== Databases: meaning and representation of facts ===

Lecture material:


* [[Media:Kr_plain_data_db_logic.ppt | relation of plain data in databases to logic & representing complex structures in databases ]]

* [[Media:Kr_rdf_schemalessdb_3.pdf|key/value pairs and rdf, compared to the relational model]]

See also about databases and logic:

* [http://www.cs.rice.edu/~tlogic/Database/all-lectures.pdf Logic and Database Queries (readable)]
* [http://magenta.ruc.dk/dat_en/research/reports/127.pdf logic in databases workshop (complex stuff)]

See also about RDF and related:

* [http://cs.ttu.ee/kursused/wav4130/rdftutapp.pdf rdf tutorial addendum for logic and dbases]
* [http://en.wikipedia.org/wiki/Resource_Description_Framework RDF wikipedia]
* [http://www.w3.org/TR/rdf-primer/ w3c RDF primer]
* [http://en.wikipedia.org/wiki/RDFa wiki intro]
* [[Media:portaalidekoosvoime.ppt]] or as pdf [[Media:portaalidekoosvoime.pdf]]
* [http://www.w3.org/2006/07/SWD/RDFa/primer/ w3c rdfa primer]
* [http://tripletalk.wordpress.com/2011/01/25/rdfa-deployment-across-the-web/ rdfa and microformats deployment now]
* [http://www.microformats.org microformats: a similar technology]
* [http://www.w3.org/TR/2003/NOTE-lbase-20031010/ Hayes & Guha Lbase]

=== Rules in logic: apps in planning, robotics, ... ===

Lecture material:
* [[Media:Kr_rules1.pdf|Rules and logic: refresher]] and as [[Media:Kr_rules1.ppt|ppt]]

Topics:

* propositional, first order, higher order
* semantics/models
* proofs and deriving new information
* automated reasoning

See also:

* [[Media:Geoffreasoningnotes.pdf|Geoff reasoning course notes: just for background, no questions in exam]]
* [http://www.ee.pdx.edu/~mperkows/May4/005.Ch11-planning.ppt planning and blocks world]
* [[blocks world axiom/query examples from lecture]]

=== Various rule/logic languages ===

Lecture material:
* [[Media:Kr_rdfs.pdf|RDFS: rdf schema]] and as [[Media:Kr_rdfs.ppt|ppt]]
* [[Notes about RDF and OWL: logical meaning]]

Several languages:
* RDFS
* OWL
* KIF
* CL
* ontologies
* wordnet
* cyc
* restricted english systems

See also:
* [http://en.wikipedia.org/wiki/Resource_Description_Framework RDF wikipedia]
* [http://www.w3.org/TR/rdf-primer/ w3c RDF primer]
* [http://en.wikipedia.org/wiki/Web_Ontology_Language OWL wikipedia]
* [http://www.w3.org/TR/owl-guide/ w3c OWL guide]
* [http://cl.tamu.edu/ CL main page],
* [http://attempto.ifi.unizh.ch/site/pubs/papers/phd_kaljurand.pdf Attempto restricted english]

=== Scraping semistructured data ===

* Example for background: http://www.sightsmap.com/

=== Time, context, metainformation ===

=== Probabilistic and nonmonotonic reasoning ===

See also:
* [http://plato.stanford.edu/entries/logic-nonmonotonic/ nonmonotonic logic]
* [http://en.wikipedia.org/wiki/Default_logic default logic]

=== Personalized search and recommendations ===

=== Fast search of facts and rules using indexing ===

* [http://www.lambda.ee/index.php/Pilt:IndexNotes04.ppt G. Molina lecture 4]
* [http://www.lambda.ee/index.php/Pilt:IndexNotes05.ppt G. Molina lecture 5]
* [http://www.lambda.ee/index.php/Pilt:Fulltext_indexes.ppt good intro and overview]
* [http://www.lambda.ee/index.php/Pilt:Mccuneindexing.pdf mccune paper about term indexes]

=== Learning ===

See also:

* [http://www.cs.toronto.edu/~hinton/csc2515/notes/lec1.ppt intro]
* [http://gamma.cs.unc.edu/rob-course/public_html/LEC/MachineLearningMethods.ppt robots/cars]
* [http://www.cs.toronto.edu/~hinton/csc2515/notes/lec4.ppt backpropagation]
* [http://www.cs.toronto.edu/~hinton/csc2515/lectures.html good set of lecture materials]
* [http://www.cs.cmu.edu/~tom/mlbook-chapter-slides.html another good full set of lecture materials]
** sh [http://www.cs.cmu.edu/afs/cs.cmu.edu/project/theo-20/www/mlbook/ch4.pdf neural netw chapter]





[[Category:Kursus]]