The voice-recognition technologies that widespread in consumer technologies have plenty of room for improvement, but the private companies that own the most common dialog systems keep their usage data private. This project was developed to make user data available to researchers who will use it to evaluate, improve, and develop new dialog and voice recognition systems.
Dialog systems like Siri and Alexa have become household names. But as any of their users know, they are far from perfect and fairly limited in their capabilities. To improve dialog systems, researchers need significant amounts of data on how consumers use them. But the companies that create dialogue systems often keep this data to themselves, making it harder for researchers to create, improve, and evaluate such systems properly.
DialPort was created with the goal of gathering data from real users for dialog systems around the world, making it available to researchers without private corporate restrictions. In addition, the DialPort DialogEcosystem supports researchers with access to tools for creating dialog systems and tutorials on how to use them. The DialogEcosystem also lets researchers crowdsource testing of dialog systems they have already developed, providing easy task creation and connection to major crowdsourcing sites. And to lower the barrier to entry to the field, the DialPort DialogEcosystem helps train young students with its REAL Challenge in which students can imagine ideal dialog systems and learn how to create them. The results from this project will ultimately impact every person who uses dialog systems in daily life.
The DialPort Portal allows dialog researchers to collect data from real users. Users come to the Portal website and are randomly paired with a dialog system. They can have a back-and-forth interaction with the dialog system. At any point, users are able to provide feedback to the system developers. For more information on using the DialPort Portal, please visit http://dialport.org/portal.html.
DialCrowd is a crowdsourcing tool that provides guidance for high-quality data collection. It reduces the time spent by requesters on more trivial aspects of creating the task with a straightforward, guided interface that results in a cleanly organized task for the worker. Quality control tasks such as duplicated tasks and golden data are added so requesters can easily compare workers’ results to a standard. Other quality tools include flagging work based on time, patterns, and agreement. All in all, DialCrowd aims to provide a seamless, positive experience for both the requester and the worker for higher quality tasks and data. For more information on using DialCrowd, please visit http://dialport.org/dialcrowd.html
For additional resources and a guide to getting started, please visit https://dialport.ict.usc.edu/.
- Zhao, Tiancheng, Kyusong Lee, and Maxine Eskenazi. "Dialport: Connecting the spoken dialog research community to real user data." 2016 IEEE Spoken Language Technology Workshop (SLT). IEEE, 2016.
- Zhao, Tiancheng, Kyusong Lee, and Maxine Eskenazi. "The dialport portal: Grouping diverse types of spoken dialog systems." Workshop on Chatbots and conversational agents. 2016.
- Lee, Kyusong, et al. "Dialport, gone live: An update after a year of development." Proceedings of the 18th annual SIGdial meeting on discourse and dialogue. 2017.
- Lee, Kyusong, et al. "DialCrowd: A toolkit for easy dialog system assessment." Proceedings of the 19th Annual SIGdial Meeting on Discourse and Dialogue. 2018.
- Lee, Kyusong, et al. "An assessment framework for dialport." Advanced Social Interaction with Agents. Springer, Cham, 2019. 79-85.
- Gunasekara, Chulaka, et al. "Overview of the ninth dialog system technology challenge: Dstc9." DSTC9 Workshop at AAAI 2021.
- Li, Zekang, Zongjia Li, Jinchao Zhang, Yang Feng, and Jie Zhou. "WeChat AI's Submission for DSTC9 Interactive Dialogue Evaluation Track." DSTC9 Workshop at AAAI 2021.
- Bao, Siqi, Bingjin Chen, Huang He, Xin Tian, Han Zhou, Fan Wang, Hua Wu, Haifeng Wang, Wenquan Wu, and Yingzhan Lin. "A Unified Pre-training Framework for Conversational AI." DSTC9 Workshop at AAAI 2021.
- Mehri, Shikib, Yulan Feng, Carla Gordon, Seyed Hossen Alavi, David Traum and Maxine Eskenazi. "Interactive Evaluation of Dialog Track at DSTC9." DSTC9 Workshop at AAAI 2021
Software Communities
National Science Foundation