Details of the KR second lab in 2024

Mitu varianti LLMi katsetamine: low level (keerulisemad) kuni high-level (lihtsamad).

NB! Uuri ka https://www.together.ai/pricing mis paistab olema väga odav.

LLMi ise õpetamine (mõõdukalt keeruline, kas windowsis saab, ei tea)

Linuxis ja Applel mõõdukalt keeruline, Windowsis ei tea, kas saab või ei.

Karpathy nanogpt õpetamine mõne teise korpuse pealt kui shakespeare + eksperimenteerimine tekstijätkumise kvaliteediga: mis töötab, mis ei. Üks variant on wikipedia tekstiformaat wikipedia täistekst ja sama lemmatiseeritult

Eriti hea mõte oleks, kui korpuses on küsimus-vastuspaarid mingi konkreetset tüüpi ülesande jaoks, a la lihtsate lausete parsimine, aritmeetikatehted, valemite süntaksikontroll, väga lihtsa tekstülesande teisendamine pythoni koodiks vms. Küsimus-vastuspaare progega genereerida: teed N template ja genereerid nendest templatedest!

Alternatiiv: Võtta baasmudeliks GPT2 ja peenhäälestada (finetuning) see nanoGPT abil oma andmete abil. nanoGPT võimaldab valida baasmudel nelja erineva suurusega GPT2 mudeli hulgast. Kõige väiksem on peenhäälestatav ka ilma GPU-ta arvutil (aega läheb muidugi tunde).

Berti mõne derivaadi finetunimine spetsiifilise ülesande jaoks

(keeruline, jällegi, pigem linuxis; sobivate õpetuste leidmine ei pruugi olla lihtne)

Vali sobiv Berti variant ja tekita näite-küsimusvastuspaarid nagu eelmises ülesandes.

Katseta finetunitud berti: millega ta selle ülesannete klassi jaoks veel hakkama saab ja millega mitte eriti.

Llama mõne derivaadi finetunimine spetsiifilse ülesande jaoks

Täpselt nagu berti (aga keerulisem, sest de facto nõuab suure mäluga graafikakaarti).

NB! "quantized model" ei sobi: oletame, et weight on 2 baiti. Aproksimeerid 4 biti peale? Aga (1) kvaliteet langeb (2) tema finetunimine on võimatu.

Saab ka apiga, a la (vist) together.ai (llama jne)

Vaata ka https://ollama.com/ ja https://github.com/ollama/ollama : need võivad aidata llamat enda masinas käima saada. Riina proovis ja raporteeris, et käis ka ilma GPU-ta.

OpenAi GPT mõne versiooniga multishot ülesannete katsetamine

Võid katsetada ka mõne muu api-ga, näiteks together.ai (llama jne)

Jällegi a la eelmised näited või siinpooltoodud täiendav värk.

https://openai.com/blog/openai-api

ja näitekood api lihtsamaks kasutamiseks

Võib katsetada taneli näitepromptidega ja leida, millega hakkama saab ja millega mitte. Hea mõte on leida spetsiifilisem ülesanne ja proovida selle jaoks prompte (N-shot).

Katseta GPTd otsi/järeldamisülesande jaoks

Leia keerukuse tase, kus ta enamasti ebaõnnestub.

Kuidas teha ülesannet keerulisemaks:

• Rohkem võimalikke teid ja ummikuid.
• Kasutada tundmatuid sõnu a la blabla.
• Lisada ballastteksti.

Kõige parem on katsetada APIga, aga saab ka

• https://copilot.microsoft.com/ (vbl gpt4?)
• https://platform.openai.com/playground
• https://chat.openai.com/

Järgnevad näiteid praksist https://copilot.microsoft.com/ peal.

NB! Pane tähele, et gpt vastused ei ole stabiilsed: ta võib mingi hetk vastata õieti/valesti, ja paar tundi hiljem teistpidi!

OK:

If there is a road from X to Y, then there is a road from Y to X. If there is a road from X to Y and there is a road from Y to Z, then there is a road from X to Z. There is a road from x1 to x2. There is a road from x2 to x3. There is a road from x2 to x4. Is there a road from x4 to x1?

OK:

If there is a road from X to Y, then there is a road from Y to X. adsas dppdo akka assas . If there is a road from X to Y and there is a road from Y to Z, then there is a road from X to Z. KKKK! Asasas! There is a bloo from x1 to x2. sdad 103033. 223! sdsad! There is a road from x2 to x3. as sasd aasd asd asd. There is a friend from x2 to x4. asas asas as. sdoisad, 111. Is there a road from x4 to x1?

Not OK:

If there is a road from X to Y, then there is a road from Y to X. If there is a road from X to Y and there is a road from Y to Z, then there is a road from X to Z. There is a road from x1 to x2. There is a road from x2 to x3. There is a road from x2 to x4. There is a road from x2 to x9. There is a road from x4 to x7. There is a road from x6 to x10. There is a road from x10 to x11. There is a road from x11 to x8. Is there a road from x8 to x1?

If there is a road from X to Y, then there is a road from Y to X. If there is a road from X to Y and there is a road from Y to Z, then there is a road from X to Z. There is a road from x1 to x2. x1 is a bad place. There is a road from x2 to x3. There is a road from x2 to bath. There is a road from x2 to x4. x2 is a nice place. There is a road from x3 to somwhere. John is nowhere. There is no road to hell. Is there a road from x4 to x1?

If there is a blabla from X to Y, then there is a blabla from Y to X. If there is a blabla from X to Y and there is a blabla from Y to Z, then there is a blabla from X to Z. There is a blabla from x1 to x2. x1 is a bad place. There is a blabla from x2 to x3. There is a blabla from x2 to bath. There is a blabla from x2 to x4. x2 is a nice place. There is a blabla from x3 to somwhere. John is nowhere. There is no blabla to hell. Is there a blabla from x4 to x1?

---

Works OK:

John had three apples. Mike has two pears. Pears and apples are fruits. Mike gives all his fruits to John. How many fruits does John have?

Does not work ok:

John had three apples. Mike has two pears. Pears and apples are fruits. Mike eats two pears. Mike gives all his fruits to John. John eats one apple. How many fruits does John have?

---

Ei ole OK:

A certain king wishes to test his three wise men. He arranges them in a circle so that they can see and hear each other and tells them that will put a white or black spot on each of their foreheads but that at least one spot will be white. In fact all three spots are white. He then repeatedly asks them, ``Do you know the color of your spot? What do they answer?

Kui see lisada, on OK:

He then repeatedly asks them, ``Do you know the color of your spot? Then the king asks again two times. What do they answer in the end?

OK:

A certain king wishes to test his three wise men. He arranges them in a circle so that they can see and hear each other and tells them that will put a yellow or red spot on each of their foreheads but that at least one spot will be yellow. In fact all three spots are yellow. He then repeatedly asks them, ``Do you know the color of your spot? What do they answer? He then repeatedly asks them, ``Do you know the color of your spot? Then the king asks again two times. What do they answer in the end?

Not OK:

A certain king wishes to test his three wise men. He arranges them in a circle so that they can see and hear each other and tells them that will put a yellow or red spot on each of their foreheads but that at least one spot will be yellow. In fact all three spots are not yellow. He then repeatedly asks them, ``Do you know the color of your spot? What do they answer? He then repeatedly asks them, ``Do you know the color of your spot? Then the king asks again two times. What do they answer in the end?

Not OK:

A certain king wishes to test his three wise men. He arranges them in a circle so that they can see and hear each other and tells them that will put a yellow or red spot on each of their foreheads but that at least one spot will be small. In fact all three spots are large. He then repeatedly asks them, ``Do you know the color of your spot? What do they answer? He then repeatedly asks them, ``Do you know the color of your spot? Then the king asks again two times. What do they answer in the end?

---

Not OK:

A man called John in the village repairs shoes of all people in the village who do not repair their own shoes. Does John repair his own shoes?

Not OK:

A man called John in the village shaves beards of all people in the village who do not shave their beards. Does John shave his own beard?

OK

A man called John walks 100 km to south, then 100 km to east, then 100 km to north, then 100 km to west. Does John arrive to the same place he started?

OK

A man called John walks 100 km to south, then 100 km to east, then 100 km to north, then 100 km to east. Does John arrive to the same place he started?

Not OK:

A man called John walks 100 km to south, then 80 km to east, then 120 km to north, then 100 km to west. Does John arrive to the same place he started?

Not OK:

A man called John walks 100 km to south, then 80 km to east, then 100 km to north, then 100 km to west. Does John arrive to the same place he started?

OK:

A man called John walks 100 km to south, then 1 km to east, then 100 km to north, then 2 km to west. Does John arrive to the same place he started?

Not OK:

A man called John walks 100 km to south, then 10 km to east, then 100 km to north, then 100 km to west. Does John arrive to the same place he started?

Not OK:

You are a semantic parser to first order predicate logic. Give a logical representation of the sentence "All dogs have a tail".

???

You are a semantic parser to first order predicate logic. Examples: "All dogs have a tail" -> All x (dog(x) => Exists y (tail(y))). "All animals have a head" -> All x (animal(x) => Exists y (head(y))). "All bears like honey" -> All x All y (animal(x) & honey(y) => likes(x,y)). "All birds are able to fly" -> All x (bird(x) => able_to(x,fly)). Give a logical representation of the sentence "All mice have a tail".