Project Details

Initially, they used third-party cloud services to run their LLM models, but it was expensive. We assisted them in moving the entire setup locally. We acquired our own GPU, set up our own lab, and now no longer rely on third-party cloud services to train our LLM models.

System Design

Data-Flow(User):

A. User wanted to load their own models

1. Go to my models
2. Click on add model
3. Enter docker image link
4. Input storage needs, opened ports, etc
5. Save model

Next

It will show in my models, select model select GPU and deploy

B. User wants to deploy a predefined model

1. User selects model