Planet SoC

KEY ACCOMPLISHMENTS LAST WEEK

• Refactored and cleaned up source code.
• Updated JavaDoc.
• Uploaded source code to code.google.com.

KEY TASKS THAT STALLED LAST WEEK

• Tried to upload source code to launchpad, but code.google.com was easier to use.

KEY CONCERNS

• None

TASKS IN THE UPCOMING WEEK

• Keep working on the project until the semester starts (Sept 2nd), at least.
• Integrate all server side processing scripts and tools (Java, Perl, R, HTML).
• Collect and analyze MySQL runtime information.

The Skoll Client retrieves a set of commands from the Skoll Server to compile and test MySQL; these commands are UNIX shell commands. My Google Summer of Code project is to work on the Java Skoll Client, I spent a great deal of time and effort getting these shell commands to run well in Java. Running shell commands in Java is not always straightforward, here are some techniques I learned to get the job done.

KEY ACCOMPLISHMENTS LAST WEEK

• Finished per test runtime information collection modifications.
• Collected and analyzed per test runtime information for a few suspicious configurations and tests.
• Refactored and cleaned up source code.

KEY TASKS THAT STALLED LAST WEEK

• None

KEY CONCERNS

• None

TASKS IN THE UPCOMING WEEK

• Refactor and clean up source code.
• Update JavaDoc.

In a previous post, I talked about the method Skoll used to collect MySQL runtime information for non-default configurations. At the time, the runtime information was collected after all of the MySQL tests were executed, which means the runtime information was accumulated from all tests run. There was no way to decipher how each test contributed to this accumulated runtime information. A greater degree of granularity can provide better understanding of the MySQL runtime behavior.

KEY ACCOMPLISHMENTS LAST WEEK

• Fixed some left over glitches in Skoll Client related to the Bazaar switch, mostly to deal with Java Runtime exec.
• Analyzed MySQL runtime data, but did not find clear differences between different configurations of MySQL. The analysis might need more granular runtime data, see next item.
• Modified Skoll Client to collect runtime information per MySQL test. Right now, Skoll Client runs mysql-test-run script (which executes all tests) then collects gcov runtime information for all tests.

KEY TASKS THAT STALLED LAST WEEK

• None

KEY CONCERNS

• None

TASKS IN THE UPCOMING WEEK

• Finish per test runtime information collection modifications.

In the Skoll project, we are trying to build a community-based distributed process to test MySQL. From the users perspective, the Skoll clients connect to the Skoll server and receive instructions to build MySQL in a specific configuration. The Skoll client then compiles MySQL and runs a set of about 750 standard MySQL installation tests. Finally, the client sends a summary of the test results back to the Skoll server. What the users do not see is the big picture. How does the Skoll server model the MySQL configuration space? How does the Skoll server select specific configurations from this space to be tested.

KEY ACCOMPLISHMENTS LAST WEEK

• Fixed Skoll Client to work with the new commands related to Bazaar switch. Running shell commends in Java, esp supporting both Java 1.4 and 5.0, is difficult and brittle.
• Analyzed MySQL runtime data with Weka. Most of the work is in retrieving/preparing the large amount of data efficiently from the database.

KEY TASKS THAT STALLED LAST WEEK

• Still did not collect runtime data with Skoll Client due to Bazaar switch.

KEY CONCERNS

• None

TASKS IN THE UPCOMING WEEK

• Runtime data collection.
• Continue to research tools and methods to analyze runtime information.

KEY ACCOMPLISHMENTS LAST WEEK

• Read the book “Data Mining: Practical Machine Learning Tools and Techniques” to learn about features of Weka.
• Played with Weka to analyze and visualize MySQL runtime data.
• As part of the effort to use push-build and Bazaar with Skoll, found a method to set $PATH in Skoll Client. Now, commands and executables in “non-standard” locations can be specified for Skoll Client (in this case bzr).

KEY TASKS THAT STALLED LAST WEEK

• Could not test or collect runtime data with Skoll Client due to Bazaar switch.

KEY CONCERNS

• None

TASKS IN THE UPCOMING WEEK

• Continue to research tools and methods to analyze runtime information.
• Runtime data collection.

KEY ACCOMPLISHMENTS LAST WEEK

• Worked on Skoll’s data processing scripts and test result generation infrastructure.
• Modified packaging format of runtime data collection, started automated runtime information collection tasks with the new Skoll client.
• Researched tools and methods to analyze runtime information (data mining and Weka).

KEY TASKS THAT STALLED LAST WEEK

• None

KEY CONCERNS

• None

TASKS IN THE UPCOMING WEEK

• Continue to research tools and methods to analyze runtime information.
• Modify Skoll to use push-build tar balls for compilation and testing.

To understand the runtime behavior of MySQL under different configurations, Skoll needs to collect runtime data while testing MySQL builds. To accomplish this, the Skoll client takes advantage of gcov, a test coverage program that’s part of the GNU Compiler Collection (GCC). gcov collects runtime information such as how many times a line of code, a method or a file was executed, what was the coverage and much more.

The source tree of MySQL actually provides a few build scripts that enables gcov under the BUILD directory. However, these scripts build MySQL using the default configuration; not exactly what Skoll needs. Skoll client builds and tests MySQL in different configurations by passing compile-time and run-time flags to the configure script before compilation. For example

KEY ACCOMPLISHMENTS LAST WEEK

• Constructed an experimental runtime data processing program.

KEY TASKS THAT STALLED LAST WEEK

• None

KEY CONCERNS

• None

TASKS IN THE UPCOMING WEEK

• Modify Skoll to use push-build tar balls for compilation and testing.
• Modify Skoll’s data processing scripts for more detailed HTML results pages.

KEY ACCOMPLISHMENTS LAST WEEK

• Analyzed the collected runtime data from Skoll Client. I am in the process of constructing a program to process/compare the runtime data.
• Improved Skoll Client so the it does NOT have to connect to two different databases in order to collect runtime information. This improvement makes management of MySQL configurations on the server side much easier.

KEY TASKS THAT STALLED LAST WEEK

• None

KEY CONCERNS

• None

TASKS IN THE UPCOMING WEEK

• Continue with runtime data processing, and then automate this processing with scripts on the Skoll server.
• Modify Skoll to use push-build tar balls for compilation and testing.

KEY ACCOMPLISHMENTS LAST WEEK

• Set up automated runtime information collection tasks on 6 machines; These machines are compiling and testing MySQL 24/7 and uploading the results to Skoll server for future analysis.
• Modified Skoll’s upload manager and data processor to handle the separate log files collected by the new Skoll Client. The separate log files include a Skoll Client log and a set of logs for each command (e.g. configure, compile, test) in a test run. The command logs are zipped together by the client before uploading to the server.

KEY TASKS THAT STALLED LAST WEEK

• None

KEY CONCERNS

• Future progress of the project depends on have push-build tar balls.

TASKS IN THE UPCOMING WEEK

KEY ACCOMPLISHMENTS LAST WEEK

• Finished the integration of new data collection features in Skoll. The Skoll Client can now collect runtime information about MySQL while running the MySQL tests.
• Collected runtime information for MySQL compiled with different configuration flags and began analyzing the collected runtime data.

KEY TASKS THAT STALLED LAST WEEK

• Had connection problems with BitKeeper (MySQL’s source control) servers while testing the new Skoll Client.
• Skoll currently gets MySQL source code from BitKeeper, however, not every revision in the BitKeeper can compile/run perfectly. Right now I have to find a “good” revision and manually request it with the Skoll Client. I need to get access to MySQL push-build tar balls to solve this bad revision problem.

KEY CONCERNS

KEY ACCOMPLISHMENTS LAST WEEK

• Continued with HTML testing report pages modifications, the new HTML pages report all the information collected by the current Skoll Client.
• Began integrating new data collection features into Skoll on the server side, most of these changes are made in our development database.
• Read research papers on reducing testing space.

KEY TASKS THAT STALLED LAST WEEK

• None

KEY CONCERNS

• Future progress of the project depends on having push-build tar balls.

TASKS IN THE UPCOMING WEEK

• Finish the integration of new data collection features in Skoll on the server side.
• Modify Skoll’s upload manager and data processor to handle the separate log files collected by the new Skoll Client. Ideally, Skoll can generate a report page for each step of the testing process (e.g. source download, compilation, testing).

KEY ACCOMPLISHMENTS LAST WEEK

KEY ACCOMPLISHMENTS LAST WEEK

• Even though this is the start of the SoC, I have been working on MySQL testing for some time now. My research group at University of Maryland has been working on automated testing for highly configurable software, and we have been testing MySQL for awhile. Our project is called Skoll (http://www.cs.umd.edu/projects/skoll/contribute/mysql.html), which automatically tests highly configurable software on distributed computing resources.

KEY TASKS THAT STALLED LAST WEEK

• Finished all my final exams and projects last week, ready to start this project full time.

KEY CONCERNS

The project I am working on for Google Summer of Code 2008 is “A Test Scheduler for the MySQL Build Farm Initiative”. The MySQL Build Farm Initiative seeks to create an automated environment that tests MySQL in multiple configurations over a powerful, virtual computing grid provided by community member’s local machines.

Currently, MySQL developers use an internal system called push-build to test a static set of configurations across a static set of computing resources. My project will take a step towards realizing the MySQL Build Farm; modify push-build to test a dynamic, rather than static, set of configurations and computing resources.

I will do this by creating a management layer that