Blossoming Intelligence: How to Run Spring AI Locally with Ollama
Elattar Saad
Sat, 11th May 2024
Nobody can dispute that AI is here to stay. Among many of its benefits, developers are using its capability to boost their productivity. It is also planned to become accessible for a fee as a SaaS or any other service once it has gained the necessary trust from enterprises. Still, We can run pre-trained models locally and incorporate them into our current app.
In this short article, we'll look at how easy it is to create a chat bot backend powered by Spring and Olama using the llama 3 model.
TechStack
This project is built using:
- Java 21.
- Spring boot 3.2.5 with WebFlux.
- Spring AI 3.2.5.
- Ollama 0.1.36.
Ollama Setup
To install Ollama locally, you simply need to head to https://ollama.com/download and install it using the proper executable to your OS.
You check is installed by running the following command:
You can directly pull a model from Ollama Models) and run it using the ollama cli, in my case I used the llama3 model:
Let's test it out with a simple prompt:
To exist, use the command:
Talking Spring
The Spring will have the following properties:
Then is our chat package, will have a chat config bean to handle:
The last step is to create a simple Chat rest controller:
Let's try and call a GET /v1/chat with an empty prompt:
What about a simple general knowledge question:
Of course, let's ask for some code:
Finally
Using models locally with such ease and simplicity can be considered as a true added value, still, the used models must be heavily inspected.
You can find the source code on this Github Repository make sure to star it if you find it useful :))
Resources
https://spring.io/projects/spring-ai
https://docs.spring.io/spring-ai/reference/api/clients/ollama-chat.html
Read next
Getting started with native java apps with GraalVM
Native Image is a technology to ahead-of-time compile Java code to a standalone executable, called a native image. This executable includes the application classes, classes from its dependencies, runtime library classes, and statically linked native code from JDK.
Wed, 10th April 2024
Read More
Leveraging Spring Reactive, Functional Endpoints, Docker, and MongoDB
Blocking is a feature of classic servlet-based web frameworks like Spring MVC. Introduced in Spring 5, Spring WebFlux is a reactive framework that operates on servers like Netty and is completely non-blocking. Two programming paradigms are supported by Spring WebFlux. Annotations (Aspect Oriented Programming) and WebFlux.fn (Functional Programming).
Thu, 29th February 2024
Read More
NextJs meets Redux: A simple user PoC
Redux is a powerful state management library primarily used in JavaScript applications, particularly those built with frameworks like React. At its core, Redux provides a predictable state container for managing the state of an application in a more organised and centralised manner. It operates on a unidirectional data flow model, which helps in maintaining the consistency of application state and facilitates easier debugging and testing.
Thu, 15th February 2024
Read More
Monitor Spring reactive microservices with Prometheus and Grafana: a how-to guide
Micro-services monitoring is a crucial aspect of managing modern, complex software architectures. Unlike traditional monolithic applications, micro-services break down functionality into smaller, independent services that can be developed, deployed, and scaled independently.
Fri, 27th October 2023
Read More
Hands on Reactive Spring with Redis Cache and Docker support
The concept of reactive programming enables more responsive and scalable programmes by handling asynchronous data streams.
Mon, 28th August 2023
Read More