Skip to content

3D multilingual chatbot with interactive gestures, contextual conversations, persistent chat history, and personalized medical and mental health support.

Notifications You must be signed in to change notification settings

nagarajpandith/aido

Repository files navigation

Aido

🏆 Awarded the 'Most Innovative Approach' at NivHack 2023 hosted at Niveus Solutions Pvt. Ltd., Mangalore.

NOTE The deployments might be down right now as the Cloud access was given only during the hackathon.

Features

  • Virtually Interactive 3D Character with speech & hand gestures
  • Multilingual Chat Sessions in different languages
  • Follow-up Questions based on previous conversation
  • Persistent chat sessions considering chat history of user
  • Personalised medical query & mental health support
  • Multilingual Voice Chat Support

Architecture

image

Tech Stack

  • Google PaLM2 based on model chat-bison using Google vertex AI
  • Deployed both client & server on Google Cloud Run
  • Database on Cloud SQL (PostgreSQL)
  • Google OAuth
  • Deepgram STT & ElevenLabs TTS
  • CI/CD using Google Cloud source repositories & Cloud build
  • Google Translate
  • T3 Stack: Next.js, tRPC, Prisma, Typescript
  • Flask Backend API

Screenshots

4 5 6 7 8

Challenges Faced

  • No finetuning available for chat-bison model
  • It was our first time using & configuring GCP & Gitlab
  • No Direct support of CI/CD of Gitlab on Cloud Run
  • Didn't get access to Google TTS & STT due to Quota Limit
  • The TTS generation & lipsyncing had a lot of latency

Future Scope

  • MedPaLM integration for better accuracy.
  • Using 3D model for other usecases like kids' trainer, lifestyle expert etc.
  • Break the sentences according to fullstops & punctuations before processing further (to avoid latency)
  • Can extend the concept to Metaverse domain

Team Members - t3-tribe

  • Nagaraj Pandith
  • Swasthik Shetty
  • Nidheesha T

About

3D multilingual chatbot with interactive gestures, contextual conversations, persistent chat history, and personalized medical and mental health support.

Topics

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published