VAPI•2y ago

Llama3 as a LLM

Hey
Is it possible to use the self-hosted Llama-3 model as LLM in agents? If not, are you planning to add this possibility in the near future?

Ask Inkeep•4/19/24, 7:46 PM

Hi @piechos, I'm an AI assistant for Vapi. While a team member reviews this question, I'll look through relevant sources and see if I can provide suggestions. Please tag @Ask Inkeep with any follow-up questions.

Ask Inkeep•4/19/24, 7:46 PM

-----------------------------
While I generate an answer, here are some sources that might be helpful:

Frequently Asked Questions
Glossary
Introduction
Custom LLMs
-----------------------------

Ask Inkeep•4/19/24, 7:46 PM

Answer

Yes, it's possible to use a self-hosted Llama-3 model as an LLM in Vapi agents. Vapi supports using any OpenAI-compatible endpoint as the LLM, including your own servers. To use a self-hosted model like Llama-3, you would set it up as a 'Custom LLM' provider, specifying the URL to your model in your API request (1) .

Sources
[1] Custom LLMs

AAsk Inkeep ### Answer Yes, it's possible to use a self-hosted Llama-3 model as an LLM in Va...

Ask InkeepAPP•4/19/24, 7:46 PM

Glad I could be helpful. Feel free to create a new thread with any new questions you may have.

dual-salmonOP•4/19/24, 8:07 PM

Is https://replicate.com/ API compatible with that feature? If not, what LLM's hosting provider is?

Replicate

Run open-source machine learning models with a cloud API

AAsk Inkeep ### Answer Yes, it's possible to use a self-hosted Llama-3 model as an LLM in Va...

Ask InkeepAPP•4/19/24, 8:07 PM

Got it. Tagging @VapiTeam for additional help.

living-lavender•4/20/24, 8:56 AM

The feature is now in the staging area for testing. You can go ahead and try it out. Additionally, you can set up any LLM you want using the custom LLM option available in the documentation.

dual-salmonOP•4/20/24, 9:37 AM

Great! How can I access your staging area?

living-lavender•4/20/24, 10:51 AM

https://staging-dashboard.vapi.ai

dual-salmonOP•4/20/24, 4:51 PM

Thank you!

dual-salmonOP•4/21/24, 11:25 AM

Unfortunately I keep getting An error occurred in this call: pipeline-error-groq-llm-failed after around 30 seconds of conversation. For example in 09ad97bd-b21c-4ef1-832f-061b1ce5763e

Ddual-salmon Unfortunately I keep getting An error occurred in this call: pipeline-error-groq...

living-lavender•4/21/24, 11:37 AM

Currently function call are not supported in Groq that's why you are getting this issue.

dual-salmonOP•4/21/24, 1:00 PM

Is this limitation on Groq's side or Vapi's side? Ans is it temporary?

unique-blue•4/21/24, 1:41 PM

living-lavender•4/21/24, 2:22 PM

Just checked with the team, and that function call is also supported with Groq.

unique-blue•4/21/24, 2:54 PM

noice

unique-blue•4/21/24, 2:55 PM

Sahil can I help develop and test function calling with all the llms at vapi? I wrote custom functions before function calling was even a thing at earlier chatgpt days. I would really love to work on this.

Uunique-blue Sahil can I help develop and test function calling with all the llms at vapi? I ...

living-lavender•4/21/24, 3:23 PM

Sure. Please test it in the staging area and let me know if you encounter any bugs. I will then forward them to the team.

dual-salmonOP•4/22/24, 3:45 PM

So, will this issue that I faced be resolved soon and we will be able to use Llama 3 through Groq?

Ddual-salmon So, will this issue that I faced be resolved soon and we will be able to use Lla...

living-lavender•4/22/24, 3:45 PM

Yup.

dual-salmonOP•4/22/24, 3:46 PM

Awesome! Do you have a page with release notes where I can track the updates?

Ddual-salmon Awesome! Do you have a page with release notes where I can track the updates?

living-lavender•4/22/24, 3:50 PM

As of now, no, but I will be working on this as well to ensure complete transparency on the things we are currently working on and all that.

living-lavender•4/22/24, 3:51 PM

https://vapi.canny.io/

Vapi

Give feedback to the Vapi team so we can make more informed product decisions. Powered by Canny.

dual-salmonOP•4/22/24, 3:52 PM

Thansk!

dual-salmonOP•4/23/24, 9:28 AM

Ok, I see that Llama3 is available through Groq in production so I tested it but when I ask the assistant to do something, It tells me that there was an issue with calling the function without even trying to call it (I saw in make.com that scenario wasn't triggered). Here is call id 10de694a-ae28-44a7-bc02-8d3f4036b57e

dual-salmonOP•4/23/24, 9:31 AM

The same happens when I try to use Mixtral through Groq and the assistant also repeats itself many times (it is not reflected in the transcript but you can hear that in the recording). Call id - 1cb77b92-f11b-4c17-9f7b-8f652855d17d

Llama3 as a LLM

Answer

Similar Threads

Similar Threads

Answer

Similar Threads