In other news, the team behind GenieFramework has just launched their new no-code builder. Make sure to check it out: Web Applications in Julia with Genie Builder.
Implement prompt caching to eliminate latency and ensure a fluid demo experience. This strategy involves storing and quickly retrieving responses for common queries or inputs, thus avoiding the need for real-time generation during the demo. It's not about deceiving your audience but about showcasing your GenAI solution's potential without technical hitches or delays (you would optimize the latency in production use cases anyway).
There are Memoization.jl and Memoize.jl, but neither of them supports caching to disk, so you cannot restart your REPL.
I prefer to use simple Dict
and if-else statement:
# Remember the conversation via key: `hash(conversation)`
CACHE = Dict{UInt64,MyMessage}()
aigenerate(x) = last(x) # mock-up only, you would need to convert MyMessage to PromptingTools types for it to work
# Conversation 1
conv1 = [MyMessage(["I am a user"], true), MyMessage(["I am Genie"], false)]
output1 = MyMessage(["Nice to meet you, Genie!"], false)
CACHE[hash(conv1)] = output1
# Example use
conversation = conv1 # known conversation
## conversation = [MyMessage("New conversation", true)] # unknown conversation
if haskey(CACHE, hash(conversation))
@info "> Cache hit!"
output_msg = CACHE[hash(conversation)]
else
@info "> Cache miss! Generating response..."
msg = aigenerate(conversation)
# Save the response for later
CACHE[hash(conversation)] = msg
end
The beauty is that
You can decide whether to cache the whole conversation or only the last user message (keep it simple as per Tip 3!)
You can then serialize the Dict
to disk and load it back when you restart your REPL.
To make your demo even more engaging, incorporate "quick action" buttons that guide users through predefined paths or use cases.
This feature not only makes the demo more feature-rich but also ensures a smoother experience by reducing the uncertainty of open-ended interactions. Quick action buttons can be easily implemented in Stipple, enhancing the flow of your presentation and making it easier for your audience to understand the full capabilities of your GenAI solution.
Additionally, by defining these actions in advance, you can more effectively leverage prompt caching, ensuring that each demonstration runs smoothly and without delay.
If you want to see how easy it is to create a stunning UI for your GenAI demo, here's a basic example using GenieFramework's Stipple.jl.
First, install GenieFramework (PromptingTools is not required, just comment it out!) Second, run the below code in your Julia REPL (or save it to a script and run it from there). Once the server starts, it will tell you to navigate to http://127.0.0.1:8000
in your browser to see the UI (or just click on the link in the REPL).
If you have any questions, there is a dedicated Genie channel on the JuliaLang Slack and the Genie team also runs a great Discord server where you can get help!
Example:
module App
using PromptingTools
using GenieFramework # GenieFramework v2.1.0
@genietools
# ! Params
GENIE_IMG = "https://easydrawingguides.com/wp-content/uploads/2021/10/how-to-draw-genie-from-aladdin-featured-image-1200.png"
INTRO_MESSAGE = [
"Welcome back, Jan!",
"What can I help you with today? Eg, `example ABC`",
]
### Helpful functions
"MyMessage is a struct that represents a message in the chat"
@kwdef struct MyMessage
id::Int = rand(Int)
name::String = "Genie"
avatar::Union{String,Nothing} = nothing
text::AbstractVector{<:AbstractString} = String[]
from_user::Bool = false
end
"Create a `MyMessage` from a user or from Genie"
function MyMessage(text::AbstractVector{<:AbstractString}, from_user::Bool=false)
MyMessage(; from_user, text, avatar=from_user ? nothing : GENIE_IMG, name=from_user ? "me" : "Genie")
end
MyMessage(text::AbstractString, from_user::Bool=false) = MyMessage([text], from_user)
### Dashboard logic
@appname MyDemoApp
@app begin
@in btn_send = false
@in user_input = ""
@in conversation = MyMessage[]
@onchange isready begin
@info "> Dashboard is ready"
conversation = [MyMessage(INTRO_MESSAGE, false)]
end
@onbutton btn_send begin
@info "> User said: $user_input" # for tracking in REPL
# Easy way to reset conversation -> just send "reset"
if strip(lowercase(user_input)) == "reset"
user_input, conversation = "", [MyMessage(INTRO_MESSAGE, false)]
elseif !isempty(user_input)
## New converation message
conversation = push!(conversation, MyMessage(user_input, true))
# Genie's response logic goes BELOW, eg, `aigenerate(user_input)`
genie_says = "Hey... I'm still learning. I don't know how to respond to that yet."
user_input = "" # empty the user input
conversation = push!(conversation, MyMessage(genie_says, false))
end
end
end
### Dashboard UI
function ui()
[
heading("My First Genie Demo"),
## Row 1: Chat
row(class="",
[
cell(class="st-module",
[
## awesome trick that allows to pass a vector of messages (=`conversation`) and generates an object for each
chatmessage(R"message.text", name=R"message.name", sent=R"message.from_user", avatar=R"message.avatar", size=4, @for("message in conversation"), key=R"message.id"),
]),
]),
## Row 2: Input From User
row([
cell(class="st-module",
[
Html.div(class="input-group",
[
textfield("Waiting for your requests... Try: `<example command>`",
:user_input,
@on("keyup.enter", "btn_send = !btn_send")),
btn("Send", @click(:btn_send)),
])]),
]),
]
end
@page("/", ui)
# Start the server
Genie.isrunning(:webserver) || up()
end # end of module