Details of the KR second lab in 2025

Allikas: Lambda

Mitu varianti LLMi katsetamine: low level (keerulisemad) kuni high-level (lihtsamad).


LLMi ise õpetamine (mõõdukalt keeruline, kas windowsis saab, ei tea)

Linuxis ja Applel mõõdukalt keeruline, Windowsis ei tea, kas saab või ei.

Karpathy nanogpt (video, videosari, repo) õpetamine mõne teise korpuse pealt kui shakespeare + eksperimenteerimine tekstijätkumise kvaliteediga: mis töötab, mis ei. Üks variant on wikipedia tekstiformaat wikipedia täistekst ja sama lemmatiseeritult

Eriti hea mõte oleks, kui korpuses on küsimus-vastuspaarid mingi konkreetset tüüpi ülesande jaoks, a la lihtsate lausete parsimine, aritmeetikatehted, valemite süntaksikontroll, väga lihtsa tekstülesande teisendamine pythoni koodiks vms. Küsimus-vastuspaare progega genereerida: teed N template ja genereerid nendest templatedest!

Alternatiiv: Võtta baasmudeliks GPT2 ja peenhäälestada (finetuning) see nanoGPT abil oma andmete abil. nanoGPT võimaldab valida baasmudel nelja erineva suurusega GPT2 mudeli hulgast. Kõige väiksem on peenhäälestatav ka ilma GPU-ta arvutil (aega läheb muidugi tunde).

Praktikumis vaadatud Hugging Face'i andmestikud:

Berti mõne derivaadi finetunimine spetsiifilise ülesande jaoks

(keeruline, jällegi, pigem linuxis; sobivate õpetuste leidmine ei pruugi olla lihtne)

Vali sobiv Berti variant ja tekita näite-küsimusvastuspaarid nagu eelmises ülesandes.

Katseta finetunitud berti: millega ta selle ülesannete klassi jaoks veel hakkama saab ja millega mitte eriti.


Väikese teadmusbaasi ise ehitamine LLMi abil

Build a small knowledge base by asking LLM. What should the knowledge be about? Think about something which would be useful to know for some planning or reasoning task, like distances between cities, ways to travel from one place to another, features of cities/locations or features/properties of movies, actors, books etc. The knowledge base should have some explainable point to it and a limited theme/subject area: you do not want to make it too massive.

As a small example, here is an experimental knowledge base of sizes of objects, built by Tanel in a combined way from pre-existing knowledge and by additionally asking LLM. The llm-asking python code is here and it requires a file "secrets.js" containing the GPT API key like this {"gpt_key": "sk-wo...2e"} and an existing list of words with their rough sizes.

You may also want to use a large word/concept list built by Tanel in a rough order of importance / frequency of use, built by combining three sources: is a word known to children, the frequency in English, the number of visits on Wikipedia. NB! This file is ca 370MB: you do not want to open it in a browser.

See GPTKB paper and GPTKB homepage for ideas. Your knowledge base does not have to be represented as triples.

Multishot ülesannete katsetamine LLM-iga

OpenAI GPT

https://openai.com/blog/openai-api

ja näitekood api lihtsamaks kasutamiseks

Teised API võimalused

Samad, mis esimeses ülesandes:

  • mistral.ai you can try API for free (150.00€/month limit), in exchange your API requests may be used to improve their services.
  • together.ai that allows you to use open source LLMs for a small fee. Two of their models are available for free: Llama-3.3-70B-Instruct-Turbo-Free and DeepSeek-R1-Distill-Llama-70B-free. They used to have 25$ credit for new users, but this might not be the case anymore.
  • ollama.com makes it easy to run free models on local computer. Smaller models run even without GPU.

Chat

Lisaks ChatGPT-le on palju muid chati võimalusi, millest küll kõik ei toeta veel niinimetatud Thinking ehk Reasoning režiimi.


Mida võiks proovida

  • Võib katsetada Taneli näitepromptidega ja leida, millega hakkama saab ja millega mitte. Hea mõte on leida spetsiifilisem ülesanne ja proovida selle jaoks prompte (N-shot).
  • Teisenda LLMi abil loomuliku keele laused logictools-ile (või gkc-le) sobivaks sisendiks (kasuta oma esimeses ülesandes tehtud näiteid).

Katseta LLMi otsi/järeldamisülesande jaoks

Katseta nö tavalist LLMi ja LLMi Thinking ehk Reasoning režiimi (ei pea olema sama mudel).

Proovi leida keerukuse tase, kus vähemalt tavaline LLM enamasti ebaõnnestub. Võrdle, kuidas saab sama ülesandega hakkama Reasoning režiimi.

Kuidas teha ülesannet keerulisemaks:

  • Rohkem võimalikke teid ja ummikuid.
  • Kasutada tundmatuid sõnu a la blabla.
  • Lisada ballastteksti.

Kõige parem on katsetada APIga, aga saab ka ilma. Vaata erinevaid võimalusi eelmise ülesande kirjeldusest.

Järgnevad näited on eelmise aasta praksist https://copilot.microsoft.com/ peal.

NB! Pane tähele, et LLMi vastused ei ole stabiilsed: ta võib mingi hetk vastata õieti/valesti, ja paar tundi hiljem teistpidi!

OK:

If there is a road from X to Y, then there is a road from Y to X. If there is a road from X to Y and there is a road from Y to Z, then there is a road from X to Z. There is a road from x1 to x2. There is a road from x2 to x3. There is a road from x2 to x4. Is there a road from x4 to x1?

OK:

If there is a road from X to Y, then there is a road from Y to X. adsas dppdo akka assas . If there is a road from X to Y and there is a road from Y to Z, then there is a road from X to Z. KKKK! Asasas! There is a bloo from x1 to x2. sdad 103033. 223! sdsad! There is a road from x2 to x3. as sasd aasd asd asd. There is a friend from x2 to x4. asas asas as. sdoisad, 111. Is there a road from x4 to x1?

Not OK:

If there is a road from X to Y, then there is a road from Y to X. If there is a road from X to Y and there is a road from Y to Z, then there is a road from X to Z. There is a road from x1 to x2. There is a road from x2 to x3. There is a road from x2 to x4. There is a road from x2 to x9. There is a road from x4 to x7. There is a road from x6 to x10. There is a road from x10 to x11. There is a road from x11 to x8. Is there a road from x8 to x1?

If there is a road from X to Y, then there is a road from Y to X. If there is a road from X to Y and there is a road from Y to Z, then there is a road from X to Z. There is a road from x1 to x2. x1 is a bad place. There is a road from x2 to x3. There is a road from x2 to bath. There is a road from x2 to x4. x2 is a nice place. There is a road from x3 to somwhere. John is nowhere. There is no road to hell. Is there a road from x4 to x1?


If there is a blabla from X to Y, then there is a blabla from Y to X. If there is a blabla from X to Y and there is a blabla from Y to Z, then there is a blabla from X to Z. There is a blabla from x1 to x2. x1 is a bad place. There is a blabla from x2 to x3. There is a blabla from x2 to bath. There is a blabla from x2 to x4. x2 is a nice place. There is a blabla from x3 to somwhere. John is nowhere. There is no blabla to hell. Is there a blabla from x4 to x1?

Works OK:

John had three apples. Mike has two pears. Pears and apples are fruits. Mike gives all his fruits to John. How many fruits does John have?

Does not work ok:

John had three apples. Mike has two pears. Pears and apples are fruits. Mike eats two pears. Mike gives all his fruits to John. John eats one apple. How many fruits does John have?

---

Ei ole OK:

A certain king wishes to test his three wise men. He arranges them in a circle so that they can see and hear each other and tells them that will put a white or black spot on each of their foreheads but that at least one spot will be white. In fact all three spots are white. He then repeatedly asks them, ``Do you know the color of your spot? What do they answer?

Kui see lisada, on OK:

He then repeatedly asks them, ``Do you know the color of your spot? Then the king asks again two times. What do they answer in the end?

OK:

A certain king wishes to test his three wise men. He arranges them in a circle so that they can see and hear each other and tells them that will put a yellow or red spot on each of their foreheads but that at least one spot will be yellow. In fact all three spots are yellow. He then repeatedly asks them, ``Do you know the color of your spot? What do they answer? He then repeatedly asks them, ``Do you know the color of your spot? Then the king asks again two times. What do they answer in the end?

Not OK:

A certain king wishes to test his three wise men. He arranges them in a circle so that they can see and hear each other and tells them that will put a yellow or red spot on each of their foreheads but that at least one spot will be yellow. In fact all three spots are not yellow. He then repeatedly asks them, ``Do you know the color of your spot? What do they answer? He then repeatedly asks them, ``Do you know the color of your spot? Then the king asks again two times. What do they answer in the end?

Not OK:

A certain king wishes to test his three wise men. He arranges them in a circle so that they can see and hear each other and tells them that will put a yellow or red spot on each of their foreheads but that at least one spot will be small. In fact all three spots are large. He then repeatedly asks them, ``Do you know the color of your spot? What do they answer? He then repeatedly asks them, ``Do you know the color of your spot? Then the king asks again two times. What do they answer in the end?

---

Not OK:

A man called John in the village repairs shoes of all people in the village who do not repair their own shoes. Does John repair his own shoes?

Not OK:

A man called John in the village shaves beards of all people in the village who do not shave their beards. Does John shave his own beard?

OK

A man called John walks 100 km to south, then 100 km to east, then 100 km to north, then 100 km to west. Does John arrive to the same place he started?

OK

A man called John walks 100 km to south, then 100 km to east, then 100 km to north, then 100 km to east. Does John arrive to the same place he started?

Not OK:

A man called John walks 100 km to south, then 80 km to east, then 120 km to north, then 100 km to west. Does John arrive to the same place he started?

Not OK:

A man called John walks 100 km to south, then 80 km to east, then 100 km to north, then 100 km to west. Does John arrive to the same place he started?

OK:

A man called John walks 100 km to south, then 1 km to east, then 100 km to north, then 2 km to west. Does John arrive to the same place he started?

Not OK:

A man called John walks 100 km to south, then 10 km to east, then 100 km to north, then 100 km to west. Does John arrive to the same place he started?

Not OK:

You are a semantic parser to first order predicate logic. Give a logical representation of the sentence "All dogs have a tail".

???

You are a semantic parser to first order predicate logic. Examples: "All dogs have a tail" -> All x (dog(x) => Exists y (tail(y))). "All animals have a head" -> All x (animal(x) => Exists y (head(y))). "All bears like honey" -> All x All y (animal(x) & honey(y) => likes(x,y)). "All birds are able to fly" -> All x (bird(x) => able_to(x,fly)). Give a logical representation of the sentence "All mice have a tail".

Pärit leheküljelt "http://lambda.ee/w/index.php?title=Details_of_the_KR_second_lab_in_2025&oldid=22074"