Fix Browser-Only ONNX Model for Text Generation
Budget: $10 – $30 USD
I need an expert to make a browser-only ONNX model (transformers.js + onnxruntime-web) load and generate text from my own signed URLs behind Nginx + Docker—no Hugging Face / external calls.
Current symptoms (what you’ll fix)
Console shows errors like: “Unsupported model type: null”, “Failed to fetch / config fetch”, or requests to huggingface.co/.../resolve/main/... despite signed URLs being provided.
CSP blocks unexpected external fetches.
Occasionally JS tries to treat a URL as a repoId, triggering HF fallbacks.
We use signed, time-limited URLs served by our /manifest and /models/... origin.
Tech Stack (already in place)
Frontend: Vanilla JS, transformers.js (browser ESM), onnxruntime-web (WASM threaded).
Backend: Node (Express) only for static + manifest + gated signed URLs.
Reverse proxy: Nginx.
Runtime: Docker compose (app, origin, viewer).
Model files present: model.onnx, config.json, tokenizer.json, tokenizer_config.json, generation_config.json.
---
Non-Negotiable Requirements (must all be met)
1. Free generation using the browser’s hardware (client-side only)
All inference runs in the user’s browser/device.
No server-side inference or external APIs.
2. Optimized for low-end devices and all major browsers
Works on weak phones/old laptops.
WASM threading/memory adapt to available resources.
3. Strictly client-side generation
The server never executes inference.
Users can only “abuse” their own CPU/RAM; the app backend remains safe.
4. Server-side gating of model downloads
Model files are served via signed, time-limited URLs.
Prevent infinite re-download abuse; once cached locally, generation is unlimited.
5. Unique, non-repetitive questions with factually correct answers
Avoid repeated prompts/outputs.
Answers must align with real-world facts for a useful experience.
6. Independence from third-party AI providers
No Hugging Face Hub, OpenAI, Anthropic, etc.
Only VPS + domain provider are allowed dependencies.
All models/assets are self-hosted.
7. Adaptive, AI-driven difficulty that scales with user accuracy
Difficulty dynamically adjusts by reading user accuracy metrics.
Difficulty must be genuinely AI-driven (not pre-baked templates / trivial paraphrases).
8. Unaffected by user volume in public production
System performance must not degrade with high traffic.
Client-side inference scales “for free”.
Server only serves static assets + gated model downloads (both cacheable/lightweight).
---
What “Done” Looks Like (how the app should behave after your fix)
On page load, the app:
Calls /manifest, receives signed URLs for exactly these files: model.onnx, tokenizer.json, config.json (optionally tokenizer_config.json, generation_config.json).
Loads transformers.js as ESM, configures onnxruntime-web to use the shipped ort-wasm-simd-threaded.wasm path.
Builds the pipeline using only the signed URLs (no repoId strings).
Example call shape (not prescriptive):
pipeline("text-generation", null, { model: <signed .onnx>, tokenizer: <signed tokenizer.json>, config: <signed config.json>, file_type: "onnx", device: "wasm", dtype: "fp32" })
No network requests to external domains. CSP passes cleanly.
When generating:
Produces short, verifiable yes/no statements with labels (as the app UI expects).
Adapts difficulty based on local accuracy stats.
Avoids repeats (local seen-set logic).
Works reliably across refreshes (no stale cache breakage).
Nginx:
Serves app and viewer.
Proxies /models/... to origin container.
Optional HF-style /resolve/main/ rewrite maps to local /models/... paths.
Large files stream correctly; caching headers set.
Smoke tests pass:
curl https://triviall.com/healthz → ok
curl -s https://triviall.com/manifest -d '{"proposed":{"backend":"wasm"}}' | jq → valid JSON with 5 files.
Signed URLs for config.json, tokenizer.json, model.onnx return 200 via both Nginx and direct origin.
In DevTools Network panel, no requests to huggingface.co or other third parties.
Deliverables
Updated frontend code that always uses our signed URLs and never infers repo IDs.
Any minimal Nginx tweaks needed (rewrite/caching headers).
A short “How to test” snippet (console steps) to prove generation runs locally with no external calls.
PR or patch files with clear diffs.
What to include in your proposal
Real experience with transformers.js, onnxruntime-web, browser-side ONNX.
Exact plan to prevent repoId inference / HF fallbacks.
A quick outline of how you’ll validate CSP and caching.
How soon you can deliver.
Screening Questions (brief)
1. In transformers.js (browser), how do you force loading from a direct .onnx URL without a repoId and prevent any HF “resolve/main” fetches?
2. What options or environment flags (e.g., env.remoteURL, allowRemoteModels, file_type) are key to avoid safetensors/pt paths?
3. How would you configure onnxruntime-web to ensure the app uses the shipped ort-wasm-simd-threaded.wasm and reasonable thread counts on low-end devices?
4. Show a minimal code snippet that builds a pipeline from explicit URLs only (model, tokenizer, config) and generates 8 tokens.
5. What’s your strategy to make sure no network calls hit huggingface.co even if the library’s defaults change?
Current symptoms (what you’ll fix)
Console shows errors like: “Unsupported model type: null”, “Failed to fetch / config fetch”, or requests to huggingface.co/.../resolve/main/... despite signed URLs being provided.
CSP blocks unexpected external fetches.
Occasionally JS tries to treat a URL as a repoId, triggering HF fallbacks.
We use signed, time-limited URLs served by our /manifest and /models/... origin.
Tech Stack (already in place)
Frontend: Vanilla JS, transformers.js (browser ESM), onnxruntime-web (WASM threaded).
Backend: Node (Express) only for static + manifest + gated signed URLs.
Reverse proxy: Nginx.
Runtime: Docker compose (app, origin, viewer).
Model files present: model.onnx, config.json, tokenizer.json, tokenizer_config.json, generation_config.json.
---
Non-Negotiable Requirements (must all be met)
1. Free generation using the browser’s hardware (client-side only)
All inference runs in the user’s browser/device.
No server-side inference or external APIs.
2. Optimized for low-end devices and all major browsers
Works on weak phones/old laptops.
WASM threading/memory adapt to available resources.
3. Strictly client-side generation
The server never executes inference.
Users can only “abuse” their own CPU/RAM; the app backend remains safe.
4. Server-side gating of model downloads
Model files are served via signed, time-limited URLs.
Prevent infinite re-download abuse; once cached locally, generation is unlimited.
5. Unique, non-repetitive questions with factually correct answers
Avoid repeated prompts/outputs.
Answers must align with real-world facts for a useful experience.
6. Independence from third-party AI providers
No Hugging Face Hub, OpenAI, Anthropic, etc.
Only VPS + domain provider are allowed dependencies.
All models/assets are self-hosted.
7. Adaptive, AI-driven difficulty that scales with user accuracy
Difficulty dynamically adjusts by reading user accuracy metrics.
Difficulty must be genuinely AI-driven (not pre-baked templates / trivial paraphrases).
8. Unaffected by user volume in public production
System performance must not degrade with high traffic.
Client-side inference scales “for free”.
Server only serves static assets + gated model downloads (both cacheable/lightweight).
---
What “Done” Looks Like (how the app should behave after your fix)
On page load, the app:
Calls /manifest, receives signed URLs for exactly these files: model.onnx, tokenizer.json, config.json (optionally tokenizer_config.json, generation_config.json).
Loads transformers.js as ESM, configures onnxruntime-web to use the shipped ort-wasm-simd-threaded.wasm path.
Builds the pipeline using only the signed URLs (no repoId strings).
Example call shape (not prescriptive):
pipeline("text-generation", null, { model: <signed .onnx>, tokenizer: <signed tokenizer.json>, config: <signed config.json>, file_type: "onnx", device: "wasm", dtype: "fp32" })
No network requests to external domains. CSP passes cleanly.
When generating:
Produces short, verifiable yes/no statements with labels (as the app UI expects).
Adapts difficulty based on local accuracy stats.
Avoids repeats (local seen-set logic).
Works reliably across refreshes (no stale cache breakage).
Nginx:
Serves app and viewer.
Proxies /models/... to origin container.
Optional HF-style /resolve/main/ rewrite maps to local /models/... paths.
Large files stream correctly; caching headers set.
Smoke tests pass:
curl https://triviall.com/healthz → ok
curl -s https://triviall.com/manifest -d '{"proposed":{"backend":"wasm"}}' | jq → valid JSON with 5 files.
Signed URLs for config.json, tokenizer.json, model.onnx return 200 via both Nginx and direct origin.
In DevTools Network panel, no requests to huggingface.co or other third parties.
Deliverables
Updated frontend code that always uses our signed URLs and never infers repo IDs.
Any minimal Nginx tweaks needed (rewrite/caching headers).
A short “How to test” snippet (console steps) to prove generation runs locally with no external calls.
PR or patch files with clear diffs.
What to include in your proposal
Real experience with transformers.js, onnxruntime-web, browser-side ONNX.
Exact plan to prevent repoId inference / HF fallbacks.
A quick outline of how you’ll validate CSP and caching.
How soon you can deliver.
Screening Questions (brief)
1. In transformers.js (browser), how do you force loading from a direct .onnx URL without a repoId and prevent any HF “resolve/main” fetches?
2. What options or environment flags (e.g., env.remoteURL, allowRemoteModels, file_type) are key to avoid safetensors/pt paths?
3. How would you configure onnxruntime-web to ensure the app uses the shipped ort-wasm-simd-threaded.wasm and reasonable thread counts on low-end devices?
4. Show a minimal code snippet that builds a pipeline from explicit URLs only (model, tokenizer, config) and generates 8 tokens.
5. What’s your strategy to make sure no network calls hit huggingface.co even if the library’s defaults change?