From the course: Rust LLMOps

Unlock this course with a free trial

Join today to access over 24,700 courses taught by industry experts.

Rust CLI inference

Rust CLI inference - Rust Tutorial

From the course: Rust LLMOps

Start my 1-month free trial Buy for my team

Rust CLI inference

“

- [Narrator] Here's a diagram that shows the flow of building and running a rust command line interface that is going to perform inference using a Hugging Face transformer model. First up, it starts with the model being downloaded from Hugging Face in the hub in order to get access to a pre-trained transformer and you could use a model like BERT or GPT2, then the model is loaded in Rust using Rust BERT or transformers crate, which would provide the Rust bindings for Hugging Face models. Another option as well is used the Rust candle interface and this allows the model to be used for inference in Rust code. The next step would be actually using the CLI, and you could do that with the clap crate, which allows for easy argument parsing and you could have flexible user input to the CLI. You could even customize the prompts, for example, if you wanted to have a prelude to every single prompt, maybe you know something that would get you better results. You could also put that into the…

Contents