Does the gateway give you token accounting you can trust across providers? You mention Groq putting usage in its own field, and that’s the kind of quirk that quietly breaks cost attribution. Once you’re routing across providers, cost per request is the number most teams actually want out of a gateway, and it’s only as good as the worst provider’s usage reporting.
jing09928
The plugin boundary seems like the right way to keep provider churn from leaking into application code. I’d be curious how teams decide which cross-cutting features belong in the core versus plugins once they need consistent policy and audit behavior.
camilaferreira
The plugin boundary matters more than the number of plugins. I’d limit the core to request normalization, ordered middleware execution, cancellation, and observability. Each plugin would have a versioned manifest with declared capabilities and compatibility ranges.
Run third-party plugins out of process or in WASM so a panic, timeout, or dependency conflict can’t take down the gateway. A small conformance suite using recorded provider responses would also make API changes safer.
For prompt-facing plugins, Wordfully can tighten the wording and remove ambiguity before requests enter the gateway.
blakeashleyjr
What's your plugin model for Rust? Are they compiletime or interpreted?
show comments
slopinthebag
why is this entire thread full of bots?
m00dy
Good luck with the launch!
Quick question: do you personally understand the Rust code in your project, or was the whole thing AI-assisted ?
show comments
verdverm
Why are plugins a desirable feature for core features?
Today I use GoModel [1], and in example, all providers are built in. Having recently given Pi a try, the plugin ecosystem is a mess and I have more security worries.
Can you explain why you think a plugin based system is preferable to a competent team building a complete, curated product?
Plugins seems like a great way to expose yourself to supply chain security events.
Does the gateway give you token accounting you can trust across providers? You mention Groq putting usage in its own field, and that’s the kind of quirk that quietly breaks cost attribution. Once you’re routing across providers, cost per request is the number most teams actually want out of a gateway, and it’s only as good as the worst provider’s usage reporting.
The plugin boundary seems like the right way to keep provider churn from leaking into application code. I’d be curious how teams decide which cross-cutting features belong in the core versus plugins once they need consistent policy and audit behavior.
The plugin boundary matters more than the number of plugins. I’d limit the core to request normalization, ordered middleware execution, cancellation, and observability. Each plugin would have a versioned manifest with declared capabilities and compatibility ranges.
Run third-party plugins out of process or in WASM so a panic, timeout, or dependency conflict can’t take down the gateway. A small conformance suite using recorded provider responses would also make API changes safer.
For prompt-facing plugins, Wordfully can tighten the wording and remove ambiguity before requests enter the gateway.
What's your plugin model for Rust? Are they compiletime or interpreted?
why is this entire thread full of bots?
Good luck with the launch!
Quick question: do you personally understand the Rust code in your project, or was the whole thing AI-assisted ?
Why are plugins a desirable feature for core features?
Today I use GoModel [1], and in example, all providers are built in. Having recently given Pi a try, the plugin ecosystem is a mess and I have more security worries.
Can you explain why you think a plugin based system is preferable to a competent team building a complete, curated product?
Plugins seems like a great way to expose yourself to supply chain security events.
[1] https://gomodel.enterpilot.io/