Tao Xie - Research

Tao Xie - Impact Highlights

AI Dependability

An approach (in collaboration with Tencent, Ltd.) for precisely detecting issues in Neural Machine Translation (NMT) without requiring oracle translations has been deployed in both the development and the production environments at WeChat, a popular messenger app with over 1 billion monthly active users (as of March 2018). The translation service in WeChat provides tens of millions of unique translations for users of the messenger app [ICSE'19 Poster, DSN'19 Industry].
An approach (in collaboration with Microsoft Research Asia) for translating natural language sentences to regular expressions has been proposed to make the translation more robust and effective in real-world settings [EMNLP'18, NL4SE'18].

Microservice Dependability

TrainTicket (in collaboration with Fudan University), a medium-size open source benchmark microservice system, has been released to be leveraged by both industry and academia to carry out research around microservice systems. The system includes 70+ microservices (41 business microservices), including 30,000+ lines of code, along with 22 fault cases reproduced from real industry cases. Upon this system, our new techniques on debugging microservice systems have been proposed [ICSE'18 Poster, ASE'18 Short, TSE'19, TSC'19].

A blog post and follow-up blog post by Fudan CodeWidsom (in Chinese).

Software Testing

VTest (in collaboration with Alibaba), adopted and deployed both internally in Alibaba (for everyday testing of a dozen popular industrial apps, such as Taobao and Alipay with over one billion users) and externally in the Software Green Alliance to provide services for top smartphone vendors and app vendors in China [ICSE'22 SEIP].

VTest (aka MonkeyBot) was highlighted in a blog post by Alibaba (in Chinese).

WCTester (in collaboration with Tencent, Ltd.), an Android testing tool, has been deployed in daily operations for improving the quality of WeChat, a popular messenger app with over 1 billion monthly active users (as of March 2018) [FSE'16 Industry, ICSE'17 SEIP, ESEC/FSE'17 Industry, ASE'18].
Pex (in collaboration with Microsoft Research Redmond), an automated testing tool based on dynamic symbolic execution for .NET, has been shipped as part of Visual Studio 2015/2017 Enterprise Edition as IntelliTest, since April 2015. Since Sept 2008, Pex (subsequently IntelliTest) has included my contributed Fitnex path exploration technique and implementation, as one of its key underlying concepts/technologies. Pex has had high download counts in industry and academia, e.g., more than 30,000 during a 20-month period of Feb. 2008-Oct. 2009. [DSN 2009, ASE'14 Experience]
Implementation of a test prioritization technique (resulted from our collaboration with Salesforce) has been in pilot use at Salesforce. [FSE'16 Industry]
Rostra, a redundant-test detection technique, detected a high percentage (about 90%) of redundant tests among tests generated by Parasoft Jtest 4.5, a popular award-winning commercial testing tool in industry. Parasoft adopted the Rostra technique for improving Jtest’s later releases. [ASE'04]
Symstra, a technique for efficient method-sequence generation in symbolic execution, was adapted and incorporated in NASA’s Java Pathfinder testing tool and Microsoft Research’s XRT testing tool. [TACAS'05]

Software Analytics

The new field of software analytics (established in collaboration with Microsoft Research Asia) has gained increasing popularity and importance in the research community. [MALETS '11, IEEE Soft'13]. The 2022 MSR Foundational Contribution Award was given to Dongmei Zhang and Tao Xie for "for the fundamental contributions in opening new areas of software analytics research with pioneering results and techniques that have had high industrial impact."
Multiple high-impact systems of software analytics (in collaboration with Microsoft Research Asia) have been successfully deployed in daily operations of Microsoft product groups or shipped with Visual Studio, e.g., StackMine [ICSE'12, IEEE Soft'13], XIAO [ACSAC'12, ICSE'17 SEIP], SAS [DSN'14, SRDS'12, ASE'13 Experience, ICSE'14 SEIP].

In 2018, ICSE 2012 Paper on StackMine selected as the single representative paper in year 2012, and among the 20 representative papers (one paper a year) selected from over 5,000 research papers published by Microsoft Research Asia during its 20 years.
In 2015, ICSE 2012 Paper on StackMine rated among the most practically relevant software engineering research (ranked the second place) of the last five years in an industrial relevance study published in ESEC/FSE 2015.

CarStream (in collaboration with Beihang U. and CAR Inc.), a scalable, high-performance Internet-of-Vehicles (IoV) system, has been deployed to offer chauffeured car services for over 30,000 vehicles at CAR Inc. [VLDB'17 Industry, TII'17, PerCom'18] .
Mobile anlytics solutions (in collaboration with Peking U, U. Michigan, and Wandoujia Lab, a top Android App Store in China) have characterized diverse app-usage patterns by learning the behavioral data from over 17 million users, and optimizing development and release planning. [IMC'15, ICSE'16, TSE'17].

Software Security

WHYPER (in collaboration with NC State U.), the first system of Natural Language Processing (NLP) for mobile app security, has opened up the new field of text analytics for mobile security. Google has awarded a Google Faculty Research Award to support the follow-up work of WHYPER, which inspired and influenced internal related efforts inside Google. [USENIX Security'13]
ACPT (Access Control Policy Tool) (in collaboration with NIST) is available as a beta release. ACPT was covered by IEEE Intelligent System magazine, Industry Cortex, ScienceDaily, Center for Internet Security (CIS), etc. From its initial release in 2011 till May 2017, ACPT attracted 460 acquisitions from major industrial, academic, and government organizations. and this number is continuously increasing. Example early adopters of ACPT include NSA, MITRE, DISA, NOAA, SAIC, DNI, Pacific Northwest National Lab, Fermi Lab, BAE system, Lockheed Martin, Raytheon, Boeing, SMI, VA government, John Hopkins University. InfoBeyond Technology LLC has developed Security Policy Tool as a full deployable ACPT version with advanced functions and has officially released Security Policy Tool since July 1, 2017. On October 14, 2017, InfoBeyond Technology was awarded the Innovative Security Solution Award for their demonstration of Security Policy Tool at the 2017 IEEE Big Data and SDN/NFV Summit. See the Mentionable Milestones of Security Policy Tool for our ACPT research's impact. [TSP'08, POLICY'10 Demo]

Educational Software Engineering

Code Hunt (in collaboration with Microsoft Research Redmond), an educational gaming platform, has already had close to 6 million (5,981,893) users (including registered users playing on www.codehunt.com, anonymous users and accounts that access api.codehunt.com directly via the documented REST APIs) since its release in Spring 2014 till July 24 2017, for about three years. See the statistics of Code Hunt services here (a snapshot cached on July 24, 2017). The 2015 Microsoft Imagine Cup included a Code Hunt Challenge. The 2014 Microsoft Beauty of Programming Contest used Code Hunt in early rounds. The game type in Code Hunt is Coding Duel games invented by me. [ICSE'15 JSEET]
Pex4Fun (in collaboration with Microsoft Research Redmond), an educational gaming platform as precusor of Code Hunt, had gained over 1 million game-play interactions made by players around the world, since its release in Summer 2010 till Oct. 2012, for only about two years. The game type in Pex4Fun is Coding Duel games invented by me. [ICSE'13 SEE, GAS'13]

Diversity in Computing

During 2018 Fall - 2019 Spring, I served as the founding Chair of the Diversity Committee of the Department of Computer Science, University of Illinois at Urbana-Champaign.
In 2007-2018, I served as the General Chair and Program Chair of 2018 and 2017 ACM Richard Tapia Celebration of Diversity in Computing Conference (Tapia), respectively.
During 2018-2019, I served as the Initiative Mentor for the ACM SIGSOFT Initiative on Diversity and Inclusion.
During 2017 Fall - 2019 Spring, I served as a Faculty Advocate at Illinois for the FLIP (Diversifying Future Leadership in the Professoriate) Alliance, an NSF INCLUDES demonstration project with the vision to increase faculty diversity in computing at research universities by increasing the diversity of PhD graduates from the top producers of computing faculty. FLIP targets the following groups: African Americans, Hispanics, Native Americans, and People with Disabilities.
During 2017 Fall - 2019 Spring, I served as one of 7 mentors for the Illinois Academic Redshirt in Science and Engineering (ARISE) Program Sponsored by NSF
I have been mentoring formally and informally hundreds of minority students around the world via MentorNet (the E-Mentoring Network for Diversity in Engineering and Science), social media, and diversity related venues. He has established and maintained a student mentoring mailing list for minority students in computing primarily from USA, including more than 800 members as of Fall 2019.
I have supervised undergraduate/graduate research of 26 minority students during my Illinois faculty career and 19 minority students during my NC State faculty career.

Community Recognitions

I have been elevated to be an ACM Fellow (class of 2021), IEEE Fellow (class of 2018), and AAAS Fellow (class of 2019) "for contributions to software testing and analytics"
Our collaborations with Microsoft Research being highlighted in

2016 IEEE Software Blog post on "the Value of Applied Research in Software Engineering" (related Reddit discussion)
Citation for my 2016 Microsoft Research Outstanding Collaborator Award (one of 32 Microsoft Research Outstanding Collaborators from academia over the past 25 years) and related 2016 Illinois CS News
2016 ACM SIGSOFT Webinar on "Transferring Software Testing Tools to Practice" (listed on the ACM Learning Webinar web)
2016 Microsoft Research Blog post on "Microsoft Research receives IEEE Council honor"

2015 Illinois CS Click! Magazine article on "Xie’s Collaboration on Pex Benefits Microsoft and CS Education"

Media Coverage

The Next Web (TNW) 2014: Microsoft Research launches Code Hunt, an educational Web game for learning programming
MIT Technology Review 2013: Data Mining Reveals the Secret to Getting Good Answers

=============================

Software Testing: The commercial Parasoft Jtest Java testing tool won numerous awards and has been adopted by thousands of development teams worldwide. Its clients include IBM, HP, and over 10,000 companies worldwide. On 26 March 2002, Parasoft Co. announced the commercial Jtest 4.5 Java testing tool, which has been popularly used in industry. Jtest 4.5 allows the user to specify method-call lengths of generated tests from one up to three. When setting method-call lengths as three, the test generation time is long and the number of generated tests is large for relatively large programs. On January 2004, we proposed a novel definition of redundant tests based on method inputs (different from traditional definitions of redundant tests based on structural coverage) and published our preliminary results (in a technical report) on detecting a high percentage (about 90%) of redundant tests among tests generated by Jtest 4.5 and later published the more comprehensive results in our ASE 04 paper; these identified redundant tests increase the testing time without increasing the ability to detect faults or increase confidence. Parasoft announced Jtest 5.0 on 3 February 2004 and Jtest 5.1 on 28 June 2004; neither of these new versions allows the user to configure the method-call lengths for generating a large number of test inputs any more. We expect that the new techniques in our ASE 04 paper can be ultimately incorporated by the existing test generation tools to improve testing performance greatly, benefiting software practitioners.

Updates: Mar 17, 2005: Parasoft Jtest 5.1 won Software Development Magazine's 15th Annual Productivity Award.
Nov 5, 2004: Upon Parasoft's invitation, Tao Xie visited Parasoft and gave a presentation on "Strategic Automated Software Testing in the Absence of Specifications".
Sept 16, 2004: We were notified by Parasoft Co. that Parasoft Jtest 6.0 (internal version, not released yet) has fixed the test redundancy issue identified by us and added back the option to generate long method-call sequences.]

Tao Xie, Darko Marinov, and David Notkin. Rostra: A Framework for Detecting Redundant Object-Oriented Unit Tests. In Proceedings of the 19th IEEE International Conference on Automated Software Engineering (ASE 2004), Linz, Austria, pp. 196-205, September 2004. [PDF][BibTex] ©2004 IEEE

Software Testing: Within our knowledge, our ASE 03 paper is the first proposed approach of using inferred program semantic properties (other than structural coverage) for helping test generation and test selection for inspection. Automated test generation tools are able to produce and execute a large number of test inputs that extensively exercise the unit under test. However, without a priori specifications, developers need to manually verify the outputs of these test executions, which is generally impractical. To reduce this cost, we developed unit test selection techniques to select a valuable subset of automatically generated test inputs. Then developers can verify their outputs, equip them with test oracles, and put them into the existing test suite. In particular, we feed dynamically inferred program behaviors in the form of specifications (e.g., the ones inferred by Daikon) to a specification-based test generation tool (e.g., Parasoft Jtest) so that some benefits of specification-based testing can be achieved without requiring specifications. On 2 March 2004, Agitar Software Inc. announced the commercial Agitator testing tool for Java. The tool automatically generates initial tests, infers Daikon-invariant-like observations, lets developers confirm these observations to assertions, and generates more tests to violate these inferred&confirmed observations.

Updates: July 17, 2006:   Agitar people published an ISSTA 06 paper on Agitator.
                Feb 7, 2006:     Agitar CTO Alberto Savoia gave a guest lecture at Tao Xie's CSC 591T course at North Carolina State University.
                April 7, 2005:   Agitar CTO Alberto Savoia gave a guest lecture at UC Berkeley on Unit/Developer Testing, describing Agitator's philosophy.
                Mar 17, 2005:   Agitar’s Agitator and Dashboard 2.0 won Software Development Magazine's 15th Annual Jolt Product Excellence Award.
                Nov 17, 2004:   Kent Beck gave a presentation on "Developer Testing" at Developer Testing Forum. Listen to it!
                Aug 19, 2004:   Agitar CTO Alberto Savoia presented “Beyond JUnit: The Future of Developer Testing for Java” at JavaOne 2004. He attracted a standing-room only crowd of more than 600 interested developers.
                Aug 2, 2004:     Kent Beck, the creator of JUnit and pioneer of eXtreme programming joined Agitar, becoming the first Agitar Software Fellow.
                July 23, 2004: Upon Agitar's invitation, Tao Xie visited Agitar and gave a presentation on "Automated Software Testing with Inferred Program Properties".
                June 29, 2004:   Agitar’s Agitator won 2004 Duke’s Choice Award at JavaOne Developer Conference.
                Dec 6, 2003:    The tools in our ASE 03 paper were highlighted in the testdriven.com newsletter #1.

Tao Xie and David Notkin. Tool-Assisted Unit Test Selection Based on Operational Violations. In Proceedings of the 18th IEEE International Conference on Automated Software Engineering (ASE 2003), Montreal, Canada, pp. 40-48, Oct. 2003. [PDF][BibTex][Slides] (Nominated for the Best Paper Award) ©2003 IEEE. An extended version to appear in a special issue of Automated Software Engineering Journal. [PDF][BibTex]

Software Testing: Generating tests for an object-oriented program typically involves two tasks: generating relevant receiver-object states and generating relevant method arguments. Existing specification-based test generation tools such as Korat and TestEra rely on specifications such as class invariants to determine whether their directly constructed receiver-object states are valid. However, specifications often don't exist in practice. In our TACAS 05 paper, we have developed new techniques and a supporting tool, called Symstra, for indirectly constructing valid and relevant receiver-object states through method sequences, without requiring any specifications. To address the issues of state explosion and relevant-argument generation, we have defined symbolic states of object-oriented programs (produced by symbolic execution) and novel comparisons of symbolic states. Experimental results showed that the tool generates tests that achieve higher structural coverage faster than the existing tools. The Automated Software Engineering group at NASA Ames Research Center implemented a modified version of our Symstra techniques on top of Java PathFinder, a model checker and test-input generator for Java, and found the techniques to be effective in Java Pathfinder’s test-input generation. The Foundations of Software Engineering group at Microsoft Research has also implemented the Symstra techniques in their test-generation tools for C#.

Tao Xie, Darko Marinov, Wolfram Schulte, and David Notkin. Symstra: A Framework for Generating Object-Oriented Unit Tests using Symbolic Execution. In Proceedings of the 11th International Conference on Tools and Algorithms for the Construction and Analysis of Systems (TACAS 2005), Edinburgh, U.K., pp. 365-381, April 2005. [PDF][BibTex]