Advanced Multi-Lingual Search Engine Development
Budget: $30 – $250 USD
Search Engine
Build a search engine that can be started with docker compose and run a few services
- Meilisearch as the database and search engine with AI embedding
- Use infinity as the embedding api https://michaelfeil.eu/infinity/main/
- Nextjs Web UI for searching and admin screen
- Scheduler/scraper for gathering data (scrapy with python)
- Nginx web server
The scheduler/scrapper will be given a list of websites to routing gather from. These websites are primarily
- Online books, both text and pdf
- Online videos- YouTube channels (need to search transcripts, api can be used)
- Online recordings (need to search transcripts)
- Bibles (ESV, NASB, KJV, Louis Second, Darby etc)
The data would be loaded into meilisearch and routinely refreshed if data is older than a certain age and hash has changed.
We should routinely do a full scrape but only send data to meilisearch if the hash has changed. Data from meilisearch should be deleted after it passes a certain age without being refreshed.
The search should function in any language, so an open source translation model can be used to store it in other languages.
Search results should be prioritized in this order
- Results in the language used
- Bible text matches
- Other matches
Example websites to ingest:
https://bibliquest.com (French example)
https://www.stempublishing.com (text example)
Cw-archive.org (pdf examples- it embeds from google books)
biblecentre.org (mix of audio, and YouTube videos, also has many locale pages
gbv-online.org (text example)
Audio-teaching.org (audio mp3 example, these should be transcribed for search)
The Powers of the Invisible World (YouTube channel example)
https://juengerschaft.org/en/ (German example)
The only publicly available service should be nextjs, server components should be used when communicating with the search engine or scheduler
A simple admin screen can be used in nextjs to add sites to the index, request index refreshes etc, add synonyms, see progress of loading website pages to search
Search results should be a short snippet with a “show more “ button that would show a longer snippet.
Example search engines doing similar jobs
Google.com
Bibelindex.de
Second phase is to pass embedding to AI model to answering questions from the scraped data.
Build a search engine that can be started with docker compose and run a few services
- Meilisearch as the database and search engine with AI embedding
- Use infinity as the embedding api https://michaelfeil.eu/infinity/main/
- Nextjs Web UI for searching and admin screen
- Scheduler/scraper for gathering data (scrapy with python)
- Nginx web server
The scheduler/scrapper will be given a list of websites to routing gather from. These websites are primarily
- Online books, both text and pdf
- Online videos- YouTube channels (need to search transcripts, api can be used)
- Online recordings (need to search transcripts)
- Bibles (ESV, NASB, KJV, Louis Second, Darby etc)
The data would be loaded into meilisearch and routinely refreshed if data is older than a certain age and hash has changed.
We should routinely do a full scrape but only send data to meilisearch if the hash has changed. Data from meilisearch should be deleted after it passes a certain age without being refreshed.
The search should function in any language, so an open source translation model can be used to store it in other languages.
Search results should be prioritized in this order
- Results in the language used
- Bible text matches
- Other matches
Example websites to ingest:
https://bibliquest.com (French example)
https://www.stempublishing.com (text example)
Cw-archive.org (pdf examples- it embeds from google books)
biblecentre.org (mix of audio, and YouTube videos, also has many locale pages
gbv-online.org (text example)
Audio-teaching.org (audio mp3 example, these should be transcribed for search)
The Powers of the Invisible World (YouTube channel example)
https://juengerschaft.org/en/ (German example)
The only publicly available service should be nextjs, server components should be used when communicating with the search engine or scheduler
A simple admin screen can be used in nextjs to add sites to the index, request index refreshes etc, add synonyms, see progress of loading website pages to search
Search results should be a short snippet with a “show more “ button that would show a longer snippet.
Example search engines doing similar jobs
Google.com
Bibelindex.de
Second phase is to pass embedding to AI model to answering questions from the scraped data.