Audio Studio Code

A custom built IDE designed designed to translate natural language to code using speech recognition, context free grammars, and recursive decent parsing all stored on a realtime database.

Uses Firebase Realtime Database and Google Speech Recognition API

Roles:

Ethan Young & Henry Schneider
• Implemented speech recognition machine learning library
• Pre-formatted strings to put in proper format for the phrase parser
Rahul Medicharla
• Developed Custom IDE from scratch that was capable of running python
• Integrated subcomponents with Google Cloud Firebase to pass data between components
Jack Italiano
• Developed Interpreter from pre-formatted strings to recursive object representations for easy access to code.
• Developed Interpreter from objects to python syntax using recursive decent parsing.

Mission

Goal

We wanted to create a platform for voice to code that is actually useful and efficient for developers. Voice to text applications already exist, but their usefullness is questionable at best, and a gimmick at worst. We are building something in between the fun gimmicks of saying "create a ball that bounces" that will create the code for a functioning bouncing ball on the screen, and the frustratingly explicit "for int i equals 0, i less than 10, i++" to create a for loop.

Target

Our target customer is a real developer that wants to focus on the high-level design of their code rather than the low-level syntax.

Requirements

• talking should feel conversational
• you should be able to be very specific and get specific results, or very general and get general results
• the IDE should always know what you're talking about (know the scope of everything)
• you should be able to acces your code from any laptop/desktop/mobile device

Next steps:

• Create dataset for speech to psuedo code AI training
• Create a more robust interpreter
• Improve speech recognition
• Improve IDE functionality and UI/UX
• Add support for mobile devices
• Add support for user accounts

How It Works

Overview

The first step is to get speech to text data and clean the data into a standardized format. Then, this cleaned data is passed to an interpreteter that creates python objects based on the data received. These objects are stored and accessible in two ways: by line for easy navigation, and by a recursive tree structure for easy manipulation. Next, these objects are parsed into python syntax and sent to the IDE to be printed.

Data Pipeline

Speech recognition -> database input -> Interpret input to objects -> Parse objects to code -> databse output -> custom IDE

Processes

Speech Recognition

1. Utilize the python speech recognition library and the google recognition engine to process phrases from the user
2. Send each string phrase to the data base input

Database

1. put your more in depth description here

Text to Code

1. Tokenize text in the input field of database
2. Parse tokens into python objects that store all data related to the feature
• store names, values, parent, children, scope, etc.
3. Interpret python objects into proper python syntax
• Recursive decent parsing to parse objects to line by line code with all proper formatting, indentation, etc.
4. Send map of key value pairs (line_number: formatted_text) to the output field of the databse

Custom IDE

1. Event listener listens to changes in output data from database
2. outputs line by line into our custom IDE
• custom IDE is a ptyhon application with custom functions that can open, run, and save files
3. listens for RUN command where python code is compiled and ran