Experimenting with Llama 3.1 - 405B Model with 128k window size (8B and 7B)

Meta has released its latest Llama iteration, Llama 3.1, which is by far the world's largest and most capable openly available foundation model. With over 300 million total downloads of all Llama versions to date, this new release is poised to supercharge innovation and unlock opportunities for growth and exploration.

Llama 3.1 405B variation?

Llama 3.1 is a large language model that has been designed to rival the top AI models in terms of state-of-the-art capabilities in general knowledge, steerability, math, tool use, and multilingual translation. Llama 3.1 brings in huge improvements including a new model of whopping 405B parameters size.

Key Features of Llama 3.1

The latest iteration of Llama comes with several key features that make it stand out from the competition. Some of the most notable features are -

• Multilingual Support: Llama 3.1 supports multiple languages that makes it a valuable tool for businesses and organizations that operate globally.
• Long-Form Text Summarization: This model is capable of summarizing long-form text with ease (thanks to expanded 128k context window), making it an ideal tool for researchers, journalists, and students.
• Model Distillation: Llama 3.1 can distill complex models into smaller, more manageable versions, that is easier to train and deploy AI models in real-world scenarios.

Llama 3.1 has made headlines after introducing the new 405B parameter size model. This huge model will most likely not fit in on your laptop or desktop. But, we can try our hands on the smaller models i.e. 70B and 8B.

Llama 405B vs. 70B vs. 8B

The difference between the three variations is clear. The larger the model, the higher quality results. The 405B model has more knowledge about the World, so it'll most likely generate quality results compared to its smaller variations. But, 405B model cannot be easily accessed.

The model is so large that majority of its users will have to pay for the cloud infrastructure to run this model. We cannot run it on our machines, even after it's openly available for download.

We can, however, try our hands on 70B and 8B variations.

Llama 3.1 8B can be run on most modern laptops and desktops. Its smaller size and parameters make it only about 5GB in size, and requires a little over 5GB memory to load.

If you've lower than 16GB memory, try a lightweight Linux distribution such as MX Linux and use Ollama.

The memory usage of a model like Llama 3.1 (70B) depends on various factors. Here are some general estimates for memory usage -

• Small batch sizes (~1-16): Llama 3.1 (70B) typically requires around 6-12 GB of RAM per GPU.
• Medium batch sizes (~32-64): Memory usage increases to around 18-24 GB per GPU.
• Large batch sizes (~128 and above): Expect memory usage to reach up to 40-60 GB or more per GPU.

Keep in mind that these estimates are approximate and can vary depending on the specific implementation, optimization strategies, and hardware configurations used. If you're interested in precise memory usage for a particular use case, I recommend conducting experiments with your chosen hardware setup.

My experience with Llama 70B model

I tried running Llama 3.1 70B all three batch sizes on my desktop (Intel(R) Core(TM) i7-8700K CPU @ 3.70GHz 3.70 GHz, 40GB RAM, Nvidia GTX 1050 4GB), and I can report that small and medium size models loaded for me. Even though medium size model loaded in memory but it need more GPU power.

70B largest batch size could not load because it required over 40GB memory to load and 4.5GB GPU memory.

My experiment confirms that to successfully run 3.1 70B model, you must have at least 6GB GPU memory, a little over 40GB memory (just to load the model), and at least 40GB storage.

The small batch size model worked successfully for me, however, was so slow (because of large parameter size).

The 3.1 small batch size weigh around 15 GB and consumed 15.03 GB of system memory. The medium batch size is over 16 GB in size and consumed 16.30 GB on my system. Small and medium batch sizes can be run only if you've at least 20GB RAM (running Windows or a little less if you've lightweight Linux distribution) and 6 GB GPU memory.

Similarly, Llama 3.1 8B variation has three batch sizes and all three worked perfectly fine for me. You need a little over 5 GB free system memory, around 3 GB GPU memory, and 5 GB on disk.

How Does Llama 3.1 Compare to Other Models?

In terms of performance, Llama 3.1 is competitive with leading foundation models such as GPT-4, GPT-4o, and Claude 3.5 Sonnet. Its smaller models are also competitive with closed and open models that have a similar number of parameters.

Meta is planning to continue developing and improving this model in the coming months. With its advanced capabilities and multilingual support, this model is expected to become a game-changer in the world of AI.

Conclusion

Overall, the experiment concludes that even though such large models are openly available, but cannot be used on devices majority of users have. Llama 3.1 70B small batch size can be loaded on home computers but it's so slow that it's barely usable. It took around 10 seconds just to output the following line - Thank you for asking! I'm doing well, thanks to you.

That's where smaller models come into play. The 8B variation (largest batch size) will load and work perfectly find on most home computers or gaming laptops or desktops. One can train the model on their own data to get more accurate results.