Advice on Getting a Start into Research by Tao Xie

Advice on Getting a Start into Research

by Tao Xie
Tao Xie's Advice Portal: Also read Tao Xie's Advice on Writing Research Papers, Tools and Tips for Writing Papers, and Advice Collections, and Introductory Readings of Sofware Testing Research.

There are many ways of proceeding, depending on how much you have already read and done. Here are some general suggestions.

Guidelines on finding a research topic:

If you don't have specific ideas of your interests (but you are interested in pursuing research in software engineering), talk to me and I will give you an overview of different directions in software engineering/testing and the directions that I am most interested in. I will give you freedom on choosing the directions that interest you most for pursuit (in other words, I won't pick a direction or problem for you). Here is a list of research directions in software engineering (but it is a bit out of dated).

If you already have some idea of your interests (or even don't have an idea of a specific research direction but have some idea of your big research direction/topic),

Read research survey papers on that topic (e.g., introductory readings of sofware testing research, software testing survey papers, what is hot in software testing), figure out:
- what sub-directions in that topic interest you most?
- what remaining research challenges in those sub-directions?
- sometimes you may just need to focus on specific sub-directions described in those surveys and don't need to read through the whole survey (but reading widely beyond your specific research topic can be helpful too; see below for tips on generating new research topics). For example, if you want to focus on architecture-based testing, you can search the text of these testing survey papers with keywords like "architecture".
Browse the research projects on that topic and actually try out those downloadable tools (e.g. software testing projects and tools), figure out:
- what aspects do these tools test/verify/analyze unsatisfactorily the programs you collected either from open source projects or from your own projects? These aspects can become your research topics. (e.g., my work on regression-test oracle augmentation tackles one problem of existing test generation tools: they are not able to generate assertions (oracles ) but just test inputs; my work on mining API usage from search results of code search engines tackles one problem of these search engines: they do not provide a summary/high level views of the source code files returned by the search engines.)
- can you apply a tool originally for program type A (e.g., Java, C, C++ programs) on program type B (e.g., GUI applications, database applications, aspect-oriented applications, concurrent programs, spreadsheet applications, web applications, web services)? If not directly applicable, what tool adaptations are needed? These adaptations can become your research topics. (e.g., my work on testing aspect-oriented programs and testing access control policies)
- whether you can integrate different tools together and then achieve some benefits that are not available when these tools are used individually? (e.g., my work on integrating Daikon and Jtest, Csallner and Smaragdakis's work on integrating ESC/Java and JCrasher) These tool integrations can become your research topics.
- whether you can use a tool/technique originally for purpose A to achieve purpose B? (e.g., my work on using machine learning originally being used in intrusion detection to predict and avoid bugs). The application of a tool/technique on B can become your research topics.
- whether you can reduce the requirements of a tool but still achieve some benefits? (e.g., most of my work follows the theme of getting the benefits of specification-based testing in the absence of specifications)
- whether you can improve the performance of a tool by incorporating some new techniques? (e.g., my work on redundant test detection and my work on regression-test oracle augmentation)
- whether you can find interplay/synergy of static analysis and dynamic analysis? (read Mike Ernst's position paper on static and dynamic analysis, slides, slides of 99 UW/MSR summer institute, Michal Young's position paper on Symbiosis of Static Analysis and Program Testing, slides, Andreas Zeller's position paper on Program Analysis: A Hierarchy, Alex Aiken and David Evans's panel presentation slides on static/dynamic analysis, Synergy: A new algorithm for property checking, Tom Ball's paper on Testing, Abstraction, Theorem Proving: Better Together!, papers by Patrice Godefroid: Compositional Dynamic Test Generation, Software Model Checking: Searching for Computations in the Abstract or the Concrete (Talk), Software Model Checking via Static and Dynamic Program Analysis, Elsa L. Gunter and Doron Peled's Model checking, testing and verification working together, NASA's Concrete Model Checking with Abstract Matching and Refinement)
  - whether you can use symbolic execution to improve those techniques that are originally based on concrete execution? (e.g., my work on Symstra over Rostra, MSR FSE's work on Axiom Meister over Henkel&Diwan's algebraic inference tool and my Sabicu tool)
  - whether you can integrate symbolic execution and concrete execution (e.g., UIUC's CUTE tool over traditional symbolic-execution-based test generation tools)
- if you have roughly a research topic in mind, whether a tool/library can be used in your own implementation? I strongly encourage you to reuse third-party tools or libraries as much as possible in your own tool implementation and focus your research efforts on contributing to the project's parts with novelty. If a third-party tool is not available in Internet, it doesn't hurt to ask the tool author whether the tool can be made available to you.
Browse the homepages and publications of researchers on that topic (e.g., software testing researchers web, Zeller's debugging resource)
- the above guidelines on producing new research topics are still applicable here.
- read Bill Griswold's advice on how to read an engineering research paper and Gail Kaiser's uggested guidelines for finding "related work" for conference papers.
- see below for tips on collecting and compiling the bibliography on that topic.

After you have already decided your research topic (after discussing with me), you can do the following along the way of pursuing that topic:

compile a bibliography on that research topic (some samples of my bibliographies are Model Checking and Testing, Requirements Verification, Analysis, and Testing). When you compile the bibliography, try to categorize them in a meaningful way. Doing so provides two benefits: you can enjoy and reuse these links throughout your research process and the process of compiling the bibliography can make you well aware the related work. Hints on how to compile the bibliography:

search Google with keywords related to the topics (e.g., test data generation, test input generation, test generation)

if the topic is related to testing, search through my testing researchers webpage with related keywords (e.g., object-oriented, database, architecture, UML)

read latest papers on the topic and track down their references (especially those mentioned in the related work section). You can read Gail Kaiser's advice on finding "related work" for conference papers.

you may find the software engineering academic genealogy helpful in finding out more related researchers in the topic: if a researcher works in your topic, his/her adviser is very likely to work in the same area, vice versa.

Browse and keep track of proceedings of top conferences in your field and software engineering in general. You need to know what latest papers on your field are in these conferences. You can also browse and keep track of jounrals in your field and software engineering. For some papers that are not so related to your field, you don't need to read them in deailed. But once later you found out that they are relevant and you should be able to remember them and check them out. For some papers that are very relevant to your field, you should read them in a more detailed way. For a list of software engineering conferences, visit my software engineering conferences web.

put in your Internet browser's front page the links to publications webpages of those researchers who are active in your research topic. Visit them often and keep eyes on them to see whether they have new papers out. So that you can keep updated on what they are doing and see whether you can produce new research ideas by reading their latest published papers.

alternatively, you can use a web content update monitoring tool to save your frequent checks of the publications webpages. I use WebMon (download). Every morning, I open this software and "check all pages" to see whether there are any new updates on these webs. I don't keep the tool running all the time because it is not necessary for me to do the real time monitoring.

sign up some mailing lists (either academic or industrial ones) that are related to your research topics. For example, if you are working in testing topics, some sample mailing lists are junit, software-testing, model-based-testing, testdrivendevelopment, at Yahoo groups. If you are using some third-party tools or libraries, you can sign up their user or discussion mailing lists, such as those of Daikon, Alloy, BCEL, and Soot.

during the process of formulating a research problem and solutions in that research topic, you need to ask yourself the following questions (do so also when you propose an initial idea and would like to see whether it is worthwhile of investing efforts in developing/implementing the idea):

1. Is the research problem significant enough (once solved) in being practical and useful in practice? (the question is applicable only when the topic is practice-oriented; my research mostly falls into this category.)
You might have a feeling by yourself. I assume you developed software yourself in the past; then is your proposed problem really an important one based on your software development experiences? You can read Mandelin et al.'s PLDI 05 paper's first section to see how the authors came up the research problem out of their own programming experiences. To answer the question, you can also read a lot of literature or talk to different people (e.g., me, peer researchers, developers/tester in industry). The above-mentioned tip on signing up some mailing lists can help you gain a feeling by listening to people's discussion besides helping you to produce new research ideas.

2. How was the research problem addressed by the existing approaches if any? How will your proposed approach be different or better than the existing approaches (e.g., better performance, without requiring specifications, without requiring source code)?
You need to read a lot of literature and talk to different people in knowing the difference or improvement.

3. How will you evaluate your proposed approach in terms of 1). your approach addresses the research problem (in some extent); 2) your approach is better than the existing approaches if any exist?
You can see whether your evaluation can use some of the subjects for testing experiments that are collected by other researchers. In your evaluation, you can also use some open source applications, such as Daikon and Jakarta applications.
In addition, what subjects will you use in your experiments/case studies? What are the measurements you will use to reflect the performance of the approach (e.g., faster, more bugs, fewer false positives/negatives)?
Kallakuri and Elbaum's article can help you get a quick-start on doing software engineering experiments. Rothermel's class notes 18-22 on experimentations in software engineering are also useful. If you want to know more about empirical studies in general, Robert Yin's book on Case Study Research : Design and Methods is a good one. You can buy one or borrow one from me for a quick look.
The shortcut of knowing how to do a good evaluation is to read related representative papers on your topic, figure out their evaluation methodology, and try to mimic their evaluation procedure.