Could you say more about how caching works? One major advantage of sticking with a single model is saving money on cached input tokens. I'd imagine if you swap between a bunch of models, you may improve performance but cost would would balloon out of control
show comments
akshay_akula
Open source and no markup is the right default for a gateway. The caching question above is the one I would want answered before swapping models though.
show comments
ceroxylon
>The gateway adds under 1 ms for BYOK requests
Amazing! Really brilliant idea, thank you for sharing this project. There is so much ground to cover in the LLM gateway / routing / reporting world, and this is a great start. The Tinker implementation is my favorite part, fine tuning is much better than a sea of context files.
show comments
sangwook
What online signal recalibrates simulated rankings against actual task success? Also do you have a plan to support semantic caching at the router level?
show comments
swthbht
Very cool. Does your gateway decide effort levels as well? Or just models?
show comments
forgetme2020
what's the business model here. How does experiential labs make money
show comments
0xbadcafebee
You started it a week ago? I look forward to checking back in 3 weeks when you've exited for $1B
show comments
cheema33
I have not tried it yet. Is it similar to LiteLLM? If so, what sets it apart?
show comments
23david
Super interesting and congrats on the release. Curious if you initially had this in Python and then rewrote in Rust?
Could you say more about how caching works? One major advantage of sticking with a single model is saving money on cached input tokens. I'd imagine if you swap between a bunch of models, you may improve performance but cost would would balloon out of control
Open source and no markup is the right default for a gateway. The caching question above is the one I would want answered before swapping models though.
>The gateway adds under 1 ms for BYOK requests
Amazing! Really brilliant idea, thank you for sharing this project. There is so much ground to cover in the LLM gateway / routing / reporting world, and this is a great start. The Tinker implementation is my favorite part, fine tuning is much better than a sea of context files.
What online signal recalibrates simulated rankings against actual task success? Also do you have a plan to support semantic caching at the router level?
Very cool. Does your gateway decide effort levels as well? Or just models?
what's the business model here. How does experiential labs make money
You started it a week ago? I look forward to checking back in 3 weeks when you've exited for $1B
I have not tried it yet. Is it similar to LiteLLM? If so, what sets it apart?
Super interesting and congrats on the release. Curious if you initially had this in Python and then rewrote in Rust?
Finally an open source tool doing this!