Ayushman Gupta

Portfolio for MIT Media Lab

Constructionist Agent in Minecraft for AI Literacy

Multi-Agent SystemsConstructionismScaffoldingPrompt Engineering

A constructionist Agent in Minecraft that acts as a Socratic guide for students helping them form complex prompts and build structures such as houses, pyramids, etc while learning the fundamentals of mathematics and geometry.

Learning Process

The first challenge was building the complex tool calling functionality by prompting the Agent to create a detailed JSON plan which could then be parsed and executed deterministically. The next challenge was to ensure reliability of the plan which was acheived by adding a Verifier Agent and a feedback loop mechanism inspired from GAN models. Finally the biggest challenge was to ensure that students could learn the fundamentals of prompting - to do this we took the approach of scaffolding the Agent allowing for students to interact with it to build complex structures - learning along the way the details that made the fundamentals of a good prompt

Graph centric Query Answering System based on Multi-Modal data

PageRankSBERTKnowledge GraphsMulti-Modal DataPython

AI system that answers search queries by building a knowledge-graph based on multi-modal data. It involves extracting Text Data using Parallel API and video data using Youtubtube Transcripts

Learning Process

The main challeneges associated with this project were deciding how to rank different nodes which repsented entities - and due to the sparse nature of the graph I decide to use PageRank algorithm. The second challenge was determinging the relevant video contexts which involved crafting perfect queries for the Youtube API using an LLM