Software Architecture Project
Budget: $250 – $750 CAD
I am seeking a ML, Data Science, and software architect specialist to help me conceptualize and design my platform's architecture, specifically experience with embeddings and vector search.
I am building a platform where creators can share content across multiple formats and be remunerated fairly. We ingest various content, from podcasts, articles, white papers and infographics, then agglomerate it, extract the most relevant pieces of information based on the end user's profile (demographic data, parameters, queries, etc) and generate a response. Based on the user, the response may have multiple sources with various contributions to the generated answer. Using weighted calculations, we can allocate the proceeds to the content creators. Example: the podcast contributes 70% to the answer, the article 20%, and the infographic 10%. If the generated response is worth 1$, then the content creators will receive $0.7, $0.2, and $0.1 respectively. It's similar to streaming music but for various content.
Here is an approach that I’ve found: Use OpenAI's embedding and ChatGPT APIs, as well as vector databases to store the index. All ingested documentation is cleaned up and divided into smaller chunks and labeled by source. We then use the GPT embedding API to generate a vector representation of each chunk and store it in the vector db index. When a user asks a question, we convert it to an embedding, and perform an advanced semantic search for closest matches to the user's query. Then we take the most relevant chunks, included them as context along with the original question, and use the ChatGPT API to generate a response in markdown format that we then convert to HTML and display to the user. This is from docsbot.ai website.
They provide a similarity percentage from the sources they use for the generated answer, we can use that similarity percentage for the remuneration of the content creators. At this stage I need the overall architecture, then additional help to detail the semantics and the algorithms that the platform will use to process the data and provide recommendations for the end user.
I am building a platform where creators can share content across multiple formats and be remunerated fairly. We ingest various content, from podcasts, articles, white papers and infographics, then agglomerate it, extract the most relevant pieces of information based on the end user's profile (demographic data, parameters, queries, etc) and generate a response. Based on the user, the response may have multiple sources with various contributions to the generated answer. Using weighted calculations, we can allocate the proceeds to the content creators. Example: the podcast contributes 70% to the answer, the article 20%, and the infographic 10%. If the generated response is worth 1$, then the content creators will receive $0.7, $0.2, and $0.1 respectively. It's similar to streaming music but for various content.
Here is an approach that I’ve found: Use OpenAI's embedding and ChatGPT APIs, as well as vector databases to store the index. All ingested documentation is cleaned up and divided into smaller chunks and labeled by source. We then use the GPT embedding API to generate a vector representation of each chunk and store it in the vector db index. When a user asks a question, we convert it to an embedding, and perform an advanced semantic search for closest matches to the user's query. Then we take the most relevant chunks, included them as context along with the original question, and use the ChatGPT API to generate a response in markdown format that we then convert to HTML and display to the user. This is from docsbot.ai website.
They provide a similarity percentage from the sources they use for the generated answer, we can use that similarity percentage for the remuneration of the content creators. At this stage I need the overall architecture, then additional help to detail the semantics and the algorithms that the platform will use to process the data and provide recommendations for the end user.
Related categories:
Software Architecture
Machine Learning (ML)
Mathematics
Data Science
Vectorization