Rendered at 03:14:35 GMT+0000 (Coordinated Universal Time) with Cloudflare Workers.
swthbht 11 minutes ago [-]
Very cool. Does your gateway decide effort levels as well? Or just models?
Areibman 5 hours ago [-]
Could you say more about how caching works? One major advantage of sticking with a single model is saving money on cached input tokens. I'd imagine if you swap between a bunch of models, you may improve performance but cost would would balloon out of control
SilenN 5 hours ago [-]
The trick is to rarely switch, or switch at task boundaries. Often the conclusion of routing is actually "this one model is actually at the pareto front for this task, just use it always".
cameronh90 3 hours ago [-]
But then it's better to just not have a gateway switch models at all.
Just have the harness able to choose which model its sub-agents use, then tell it how to split up tasks and which models to use when doing so.
SilenN 2 hours ago [-]
That is another way to do. Or we can automatically figure out which models the subagents should be using for you. And update them as new models come out and the work your subagents do changes. More than one way to skin a cat.
purplecats 5 hours ago [-]
and caching is related to performance too ofc
akshay_akula 2 hours ago [-]
Open source and no markup is the right default for a gateway. The caching question above is the one I would want answered before swapping models though.
SilenN 2 hours ago [-]
Ans: we rarely switch, often times it's just a "switch to using this model for your agent"
ceroxylon 2 hours ago [-]
>The gateway adds under 1 ms for BYOK requests
Amazing! Really brilliant idea, thank you for sharing this project. There is so much ground to cover in the LLM gateway / routing / reporting world, and this is a great start. The Tinker implementation is my favorite part, fine tuning is much better than a sea of context files.
kfallah15 2 hours ago [-]
Thanks! We are going to add continual RL via Tinker soon too
cheema33 3 hours ago [-]
I have not tried it yet. Is it similar to LiteLLM? If so, what sets it apart?
kfallah15 3 hours ago [-]
Router and model optimization from traffic is the main differentiator
SilenN 2 hours ago [-]
Also a hosted marketplace, not just BYOK
0xbadcafebee 2 hours ago [-]
You started it a week ago? I look forward to checking back in 3 weeks when you've exited for $1B
SilenN 1 hours ago [-]
See you soon
23david 4 hours ago [-]
Super interesting and congrats on the release. Curious if you initially had this in Python and then rewrote in Rust?
SilenN 4 hours ago [-]
Yep! If you look at the commit history that's exactly what happened.
Just have the harness able to choose which model its sub-agents use, then tell it how to split up tasks and which models to use when doing so.
Amazing! Really brilliant idea, thank you for sharing this project. There is so much ground to cover in the LLM gateway / routing / reporting world, and this is a great start. The Tinker implementation is my favorite part, fine tuning is much better than a sea of context files.