Enabling children and beginning programmers to build AI programs

Click here if you wish to read this and the AI guide in a language other than English.

Introduction

The eCraft2Learn project developed a set of extensions to the Snap! programming language to enable children (and non-expert programmers) to build AI programs. The blocks are available as projects with examples of using the blocks as well as libraries to download and then import into Snap!. It is possible to run Snap! with many of the AI blocks without an Internet connection.

Important updates are documented here.

You can import any of the libraries below into your projects. If you click on the "project" link you'll open a project that illustrates usage of the library.

  1. Enabling your sprites to speak in over a hundred languages. Project or library.
  2. Enabling your sprites to listen to speech in over a hundred languages. And to recognize sounds. Project or library.
  3. Enabling your sprites to see using the camera. Project or library.
  4. Enabling your projects to do arithmetic on words and intelligently process and generate text. Project or library.
  5. Enabling your projects to create, train, and use deep learning neural networks. Project or library.
  6. Finding the nearest neighbors to a list of numbers. The neighbors vote on how to label a new list of numbers. Project or library.
  7. Miscellaneous AI blocks (style transfer, image embedding, and using Wikipedia and more). Project or library.

AI programming guides

This guide currently consists of seven chapters describing the new Snap! blocks, possible projects, sample programs, background information, and the larger context about AI and machine learning:

Adding speaking to your programs Speech synthesis requires only speakers or earphones
Adding listening to your programs Speech recognition requires a microphone (built-in or connected to a USB port)
Adding image recognition to your programs Requires a camera (built-in or connected to a USB port) and registering to get API keys
Adding pre-trained machine learning models to your programs Many examples rely upon a camera. It is very slow unless your device has a GPU.
Working with words and sentences No special hardware requirements for word embeddings. API keys needed for the rest.
Making machine learning neural nets It is very slow unless your device has a GPU.
Finding the nearest neighbors It is very slow unless your device has a GPU.

Publications

The paper AI Programming by Children by Ken Kahn and Niall Winters describes this work and was published in the Proceedings of the Constructionism 2018 Conference. AI Programming by Children using Snap! Block Programming in a Developing Country by Ken Kahn, Rani Megasari, Erna Piantari and Enjun Junaeti was published at the EC-TEL 2018 conference and describes workshops with high school students in Indonesia using these AI blocks. An early version of the AI extension to Snap! is described in the paper Child-friendly Programming Interfaces to AI Cloud Services also by Ken Kahn and Niall Winters.

Constructionism and AI: A history and possible futures by Ken Kahn and Niall Winters describes our view of the history and future of the role of AI in constructionist learning. Deep Learning Programming by All by Ken Kahn, Yu Lu, Jingjing Zhang, Niall Winters, and Gao Ming describes our efforts in enabling students to build, train, and use deep learning models (as described in Making machine learning neural nets). Both papers were published in the Proceedings of the 2020 Constructionism Conference.

Programming word embeddings in Snap! by Ken Kahn, Yu Lu, Jingjing Zhang, Niall Winters, and Ming Gao was presented at SnapCon 2019, was translated to German in for the Log in teachers magazine, and a short version appeared in the English teachers magazine Hello World. This describes the material available in the guide Working with words and language.

Learning by enhancing half-baked AI projects by Ken Kahn and Niall Winters describes many of the projects listed on this page.

AI Snap! blocks for speech input and output, computer vision, word embeddings, and neural net creation, training, and use by Naveen Naveen, Ramana Prasad, and Ken Kahn. This is the full version of the short paper in the proceedings of the Twelfth AAAI Symposium on Educational Advances in Artificial Intelligence 2022.

Presentations

This work has been presented at several conferences and universities. The slides can be found here. There are also slides from a talk about word embeddings and mathematics education.

Sample programs

The following demonstration programs use the AI Snap! blocks. They run best in the Chrome or Edge browser.

Listen to generated speech with random pitch, rate, voice, and language
(requires speaker/headphones)
Speak single word commands to a sprite (with synonym support)
(requires microphone and speaker/headphones)
Speak full sentence commands to a sprite (with keyword search)
(requires microphone and speaker/headphones)
Create funny sentences by verbally answering questions
(requires microphone and speaker/headphones)
Customise stories by verbally answering questions
(requires microphone and speaker/headphones)
Ask questions of Wikipedia
(requires microphone and speaker/headphones)
Ask about the weather anywhere in the world
(requires microphone and speaker/headphones)
Listen to a description of what is in front of the camera
 (speaker/headphones, webcam, and API key(s))
Listen to a description of what is in front of the camera in response to you speaking
(requires microphone, speaker/headphones, webcam, and API key(s))
Train the turtle to move depending on which way your finger is pointed
(requires webcam and a GPU)
Train the turtle to move depending on the voice commands you give
(requires microphone)
Train the turtle to move left or right depending on which way you lean
(requires webcam and a GPU)
Play Rock Paper Scissors using machine learning
(requires webcam, speaker/headphones and a GPU)
Play Simon Says using machine learning
(requires webcam, speaker/headphones and a GPU)
Run an app that warns you if you touch your face
(requires webcam and a GPU)
Play a game where you pop virtual balloons with your hands
(requires webcam and a GPU)
See what objects are in front of the camera
(requires webcam and a GPU)
Displays 468 3D locations on a face
(requires webcam and a GPU)
Play Guess My Word that works by turning words into 300 numbers
(requires speaker/headphones and optional microphone)
Turn a photo into 8 different paintings
(requires webcam (but easily modified to use any image instead) and a GPU)
Answers questions about this Snap! AI library using the Universal Sentence Encoder
(might be slow without a GPU)
Answers questions about Snap! using the Snap! Wikipedia article using BERT
(might be slow without a GPU)
Suggest spymaster clues for the Codenames game.
(might be slow without a GPU)
Gopher will talk with you using either GPT-3, Jurassic 1, or Cohere.
(requires an API key from OpenAI.com, studio.ai21.com, or os.cohere.ai)
Play or watch AIs play an animal guessing game
(requires an API key from OpenAI.com, studio.ai21.com, or os.cohere.ai)
Use a language model to control how a sprite moves
(requires an API key from OpenAI.com, studio.ai21.com, or os.cohere.ai - OpenAI's GPT-3 works best)
DALLE-2 can generate new costumes while your project is running
(requires an API key from OpenAI.com)
DALLE-2 can make variants and edit costumes based on your prompts
(requires an API key from OpenAI.com)
Use GPT-3 to generate stories, edit them, and suggest prompts for text-to-image apps
(requires an API key from OpenAI.com)
The GPT-3 sample project for generating illusrated stories
Using GPT-3, Jurassic 1, or Cohere to generate virtual debates
(requires an API key from OpenAI.com, AI21.com, or Cohere.ai)
The Language model sample project for generating debates on any topic
Translation using 300 numbers and a word list with 2.5% coverage
(no special requirements)
Guesses the relationship between two lists of numbers
(runs very slowly without a GPU)
Learns to predict your ratings of "Abstract Art"
(runs very slowly without a GPU)
Learns to play Tic Tac Toe
(runs very slowly without a GPU)
Names colors badly - Improve it!
(runs very slowly without a GPU)
Guesses the level of confidence indicated by a sentence
(runs very slowly without a GPU)
Tries to solve analogy problems with photos (partially successfully - can you improve it?)
(runs very slowly without a GPU)

We have collected some projects by students who have used these AI blocks.

Source code

The Snap! AI blocks load this JavaScript file on Github and then calls the functions defined therein.

Contact

Send email to Ken Kahn (toontalk@gmail.com) if you encounter problems or have questions.

GPU Requirements

A GPU is highly recommended for chapter 4 image recognition, chapter 6 machine learning, and some demos. Nearly all modern computers have a GPU. Without one or with a very primitive one programs using machine learning may run very slowly. For example, the image machine learning commands run very slowly on Raspberry Pis. One version of the audio training primitives, however, do run well on Raspberry Pis.

Using the AI blocks in your projects

You can import any of these libraries into your existing or new projects. The AI blocks can then be found in the Snap! palette under , , or any of the following custom categories