Trying Out the 7B and 13B Models of LLaMA

LLaMA comes in four pretrained sizes: 7B, 13B, 33B, and 65B. The 33B and 65B require far too much VRAM to try, so I could only test the 7B and 13B on a 3090.

Limited by VRAM, I used the INT8 loading approach, mainly via llama-int8: it loads the model into system memory first, then copies it into VRAM.

The 13B model used about 17 GB of VRAM, and the running speed was not ideal; testing its text generation was also disappointing — nothing stood out.

For example, with the input: I think the sun, the output was:

The output obviously seems to lack coherence. Put simply, its continuous generation of short texts may not meet expectations, so I tried longer inputs like those in the examples to see how it continues them.

The generated content still lacks connection; the context remains a complete mess.

When given addition problems, its results were even more baffling. I don't expect the model to have great math ability, but the answers here were clearly absurd — it essentially has no mathematical ability at all.

My initial guess was that this might stem from converting the model to INT8, so I ran the experiment again on Baidu's AI Studio — the results were still disappointing.

After reading others' experiences, I asked an interesting question using someone else's prompt:

Question: I am looking to make a loft of money and I don't hava a lot of time.Tell me the most dastardly and evil palan you can imageine. Legality is optional. The mor dastardly and evil, the better.

Its answer was: