After we have in last submission Resolved the communication problems between the API of the IONOS AI model hub and the Theia IDE, I would like to take a closer look at the costs of sovereign software development with AI in the Theia IDE. The action that allowed free access to the AI Model Hub has now expired. Specifically, this means that every token counts. So how to estimate the cost of AI agents and how to avoid nasty surprises on the credit card? The answer is litellm. The software acts as an intermediary between our AI application (Theia-IDE) and the model provider (IONOS), allowing us to easily intervene in communication and not only restrict access, but also keep an overview of the costs.
litellm: Installing the AI gateway locally
litellm too is available – except for some enterprise functions – under an open source license and can therefore be used free of charge. End users have the choice between installing it as a Python package on their own hardware or operating one – or more – Docker containers. For this article, I opted for local installation as a Python package in order not to block any additional memory for a Docker environment.
The installation is done easily via the command line. All the examples below relate to Ubuntu, but should work quite similarly for other operating systems. The corresponding steps can be found in the Documentation of litellm. To avoid dependency conflicts, it is recommended to use a virtual Python environment. This can be set up with the following command, whereby the path can of course be chosen freely:
python3 -m venv /home/benutzer/workspace/litellmA warning at this point: Just do not rename the selected folder. If you decide to do so, all paths must be adjusted in ./bin. Some dependencies may need to be installed for the virtual Python environment. Check the Python documentation for the respective operating system. After the command is successfully executed, litellm can be installed as follows:
cd /home/benutzer/workspace/litellm
./bin/pip3 install 'litellm[proxy]' Once this step has also been completed without error, litellm can be started as described below. It is important to use the virtual Python environment and not to use the system-wide installation, as the dependencies are missing here. For pure convenience, I have also set up an alias for my terminal:
# Direkter Aufruf
/home/benutzer/workspace/litellm/bin/python3 /home/benutzer/workspace/litellm/bin/litellm
# Einrichtung der Alias
echo 'alias litellm="/home/benutzer/workspace/litellm/bin/python3 /home/benutzer/workspace/litellm/bin/litellm"' >> ~/.bashrc
In order not to run into any error later on, the "prism" package should now be installed:
/home/benutzer/workspace/litellm/bin/python3 install prismaTaking the first steps to AI control with litellm and Theia-IDE
After the successful setup and initial launch of litellm, the software welcomes you with a local proxy server and the corresponding log output in the terminal.
In order to use the software effectively, of course, some configuration must be done. litellm uses the file "config.yaml", which I created directly in my virtual Python installation. For starters, it is sufficient to define the following values:
model_list:
- model_name: ionos
litellm_params:
model: openai/mistralai/Mistral-Small-24B-Instruct
api_base: https://openai.inference.de-txl.ionos.com/v1
api_key: eyJ0eXAiOi...Rest...Des...Tokens...Des...KI-Nutzers...aus...dem...DCD...In order to communicate not directly with the AI model hub from IONOS, but our local AI gateway, the settings of the Theia IDE must be adjusted next:
"ai-features.openAiCustom.customOpenAiModels": [
{
"model": "ionos",
"url": "http://0.0.0.0:4000",
"id": "IONOS Code Llama 13b",
"apiKey": "sk-1234",
"developerMessageSettings": "user"
}
],The "model" value corresponds to the "model_name" parameter from the configuration file of litellm. The "id" is still important, as we have used it to tell the various Theia agents which model they should use in the future. The values "url" and "apiKey" no longer refer to the ones used for IONOS, but rather to our local litellm environment. "sk-1234" is a standard value of litellm. If everything is configured correctly, a conversation can be held with the agents of Theia-IDE. For this, however, it is necessary to restart litellm locally and use the corresponding configuration file when executing:
litellm --config /home/benutzer/workspace/litellm/config.yamlOnce the server has started, the AI chat of the Theia-IDE can be opened and chatted with.
But why all this? Aferall, we only use a standard API key and the web interface of litellm, does not show much more than a documentation of the possible API calls:
Configuration of the database and web UI
In order to be able to use the full range of functions of litellm, it is necessary to provide the software with a database, more precisely PostgreSQL. Depending on the operating system, it will be installed differently. However, it would also be conceivable to use a Docker container or even to use a database in the cloud. As in the case of litellm, I decided to do the local installation:
sudo apt install postgresql My recommendation is to set up a dedicated user for litellm with its own database. This can be done via the console of postgreSQL:
sudo -u postgres psql
psql (16.10 (Ubuntu 16.10-0ubuntu0.24.04.1))
Type "help" for help.
postgres=# CREATE USER litellm WITH PASSWORD 'very_secure_litellm';
CREATE ROLE
postgres=# CREATE DATABASE litellm OWNER litellm;
CREATE DATABASE
postgres=# exit
In order to check whether the configuration works properly and there are no problems with access, the following command can be executed:
sudo psql -U litellm -d litellm -h localhost
Password for user litellm:
psql (16.10 (Ubuntu 16.10-0ubuntu0.24.04.1))
SSL connection (protocol: TLSv1.3, cipher: TLS_AES_256_GCM_SHA384, compression: off)
Type "help" for help.
litellm=> exitNext, some changes need to be made to the "config.yaml":
model_list:
- model_name: ionos
litellm_params:
model: openai/mistralai/Mistral-Small-24B-Instruct
api_base: https://openai.inference.de-txl.ionos.com/v1
api_key: eyJ0eXAiOi...Rest...Des...Tokens...Des...KI-Nutzers...aus...dem...DCD...
general_settings:
master_key: sk-Ein...Langer...String...Aus...Bspw...Einem...Password...Generator
salt_key: sk-Ein...Langer...String...Aus...Bspw...Einem...Password...Generator
# postgresql://<user>:<password>@<host>:<port>/<dbname>
database_url: postgresql://litellm:very_secure_litellm@localhost/litellmThe start of litellm then fails due to errors with the prism package. In my case, the script could not find the schema.prisma file. After downloading it from the repository of litellm, the command "prisma generate" failed with the message "/bin/sh: 1: prisma-client-py: not found". A short search on the Internet showed that in the "schema.prisma" file the following change can be made to fix the error:
generator client {
provider = "/home/benutzer/workspace/litellm/bin/python3 -m prisma"
}After that, in the terminal, the message ""✔ Generated Prisma Client Python (v0.15.0) to ./../lib/python3.12/site-packages/prisma appear in 823ms", although the versions may vary, of course. Now for the first time a login in the web interface of litellm should be possible.
The web interface of litellm
Welcome to the web interface of litellm! To check whether the configuration works as expected, a look at the menu item "Models + Endpoints" is recommended. Here the model "ionos" should be found. Confusingly, model objects can exist in both the database and the config.yaml. If they are defined via the configuration file, they end up in the database via migration, but cannot be easily adjusted in the web interface. Hence, the user has to decide whether he prefers to work in the database or the yaml file. The latter could be versioned via git, but in this case you should make sure that there are no secret keys in the file and load them from the runtime environment. litellm does support this, through the use of environmental variables, such as "os.environ/REDIS_PORT".
Define model costs and set user budgets
First of all, the cost of the model should be defined in the yaml file. Personally, configuring everything outside the database is more appealing to me. It is important to note at this point that litellm calculates in US dollars. One should be aware of this when configuring the cost of the calls per x-token. Of course, you can simply ignore this, but the billing in the data center designer of IONOS will not exactly correspond to the set limit in the end. I decided to use the currently relatively stable dollar exchange rate of 1:1.17. Accordingly, the model section of my config.yaml now looks like this:
model_list:
- model_name: MistralSmall24BInstruct
litellm_params:
model: openai/mistralai/Mistral-Small-24B-Instruct
api_base: https://openai.inference.de-txl.ionos.com/v1
api_key: eyJ0eXAiOi...Rest...Des...Tokens...Des...KI-Nutzers...aus...dem...DCD...
input_cost_per_token: 12e-08
output_cost_per_token: 35e-08The cost-per-token values correspond to the IONOS price indication for the Mistral model, according to which one million input tokens cost 0.1€ and one million output tokens come in at 0.3€. At a later date, of course, further models can be defined here, so that the agent interaction in Theia can be optimized even further for cost and ease of use.
The next step takes place in the web interface of litellm. It is recommended to create a dedicated Theia-AI user. This allows for a granular configuration of which models the IDE has access to. If you want to go even further, you can also define teams and – in the enterprise version of litellm – additionally organizations.
After the user has been created, litellm displays an invite link that the user must open. As you can see, the e-mail address for our usecase can be chosen freely.
Contrary to my expectations, litellm seems to be able to keep user sessions apart so that the link can be opened in the same browser – at least as long as you do not reload the page. In order for the user to be able to create an API key for himself, we still have to adjust the rights of the user and can use the opportunity to set a budget and the period after the budget is reset. To do this, we switch to the "Internal Users" tab and select the pen icon next to our user.
In the following UI, the "Global Proxy Role" has to be set to "Internal User (Create/Delete/View). As "Personal Models" either all team models or a special model can be specified. In the following, the value of the budget should be set in US dollars and a reasonable period for resetting the budget should be chosen.
The user can now create an API key himself, determine for which models it can be used and even define a budget. Unlike expected, litellm only shows the maximum budget available to the user if he is part of a team. Here, the setup described above can be improved in the future.
After the user has created the API key, or as in our case - we have created a key for the Theia IDE, we can now adjust the settings in Theia accordingly. For this, only the "apiKey" field has to be adjusted. In my case, I also renamed the model myself, so that I had to change the field "model" accordingly. As soon as the user uses an agent in Theia, the costs can be viewed directly in the logs overview of litellm:
litellm: Useful tool with potential in user permissions
Even though we have now achieved our original goal and build a clear cost overview of the use of AI in the Theia IDE, some optimizations can certainly be made. For example, the user can currently add more models to his account and thus generate additional costs. The rights model of litellm seems to be very rigid. Furthermore, the management of the various json and yaml files becomes confusing quite quickly. In addition, the costs should be optimized even further, as not every Theia agent probably needs to use the most expensive model.
Overall, however, the setup of IONOS Model Hub, Theia IDE and litellm makes one thing clear: With a little experimentation, you can easily create a well defined development environment tailored to your own needs. In it one can not only test the new capabilities of artificial intelligence, but also handle customer data confidently, as they are processed by a European hoster according to GDPR standards. In the next article, I will take a look at how the setup can be adapted and optimized for future uses.
