Planet SoC

KEY ACCOMPLISHMENTS LAST WEEK

• Refactored and cleaned up source code.
• Updated JavaDoc.
• Uploaded source code to code.google.com.

KEY TASKS THAT STALLED LAST WEEK

• Tried to upload source code to launchpad, but code.google.com was easier to use.

KEY CONCERNS

• None

TASKS IN THE UPCOMING WEEK

• Keep working on the project until the semester starts (Sept 2nd), at least.
• Integrate all server side processing scripts and tools (Java, Perl, R, HTML).
• Collect and analyze MySQL runtime information.

The GSoC is about to end up and it is time to sum up what has been done within the last weeks.

News

*Error handling added. Plugin uses the following mechanism to gather and report errors. When we can’t load/open SIGAR library or get any information from it we pass the error to user informing about that. On the other hand when only some information is not available we rather print warning about that to MySQL .err file.
Also we check for return value for each schema_store_record function.

*Plugin has been merged into the MySQL 6 server code. Now you can download and compile the MySQL server with SIGAR plugin already being there.

Downloads

Standalone SIGAR plugin is available at lp:~m.ch/mysql-server/sigar-plugin.

The Skoll Client retrieves a set of commands from the Skoll Server to compile and test MySQL; these commands are UNIX shell commands. My Google Summer of Code project is to work on the Java Skoll Client, I spent a great deal of time and effort getting these shell commands to run well in Java. Running shell commands in Java is not always straightforward, here are some techniques I learned to get the job done.

KEY ACCOMPLISHMENTS LAST WEEK

• Finished per test runtime information collection modifications.
• Collected and analyzed per test runtime information for a few suspicious configurations and tests.
• Refactored and cleaned up source code.

KEY TASKS THAT STALLED LAST WEEK

• None

KEY CONCERNS

• None

TASKS IN THE UPCOMING WEEK

• Refactor and clean up source code.
• Update JavaDoc.

In a previous post, I talked about the method Skoll used to collect MySQL runtime information for non-default configurations. At the time, the runtime information was collected after all of the MySQL tests were executed, which means the runtime information was accumulated from all tests run. There was no way to decipher how each test contributed to this accumulated runtime information. A greater degree of granularity can provide better understanding of the MySQL runtime behavior.

KEY ACCOMPLISHMENTS LAST WEEK

• Fixed some left over glitches in Skoll Client related to the Bazaar switch, mostly to deal with Java Runtime exec.
• Analyzed MySQL runtime data, but did not find clear differences between different configurations of MySQL. The analysis might need more granular runtime data, see next item.
• Modified Skoll Client to collect runtime information per MySQL test. Right now, Skoll Client runs mysql-test-run script (which executes all tests) then collects gcov runtime information for all tests.

KEY TASKS THAT STALLED LAST WEEK

• None

KEY CONCERNS

• None

TASKS IN THE UPCOMING WEEK

• Finish per test runtime information collection modifications.

In the Skoll project, we are trying to build a community-based distributed process to test MySQL. From the users perspective, the Skoll clients connect to the Skoll server and receive instructions to build MySQL in a specific configuration. The Skoll client then compiles MySQL and runs a set of about 750 standard MySQL installation tests. Finally, the client sends a summary of the test results back to the Skoll server. What the users do not see is the big picture. How does the Skoll server model the MySQL configuration space? How does the Skoll server select specific configurations from this space to be tested.

KEY ACCOMPLISHMENTS LAST WEEK

• Fixed Skoll Client to work with the new commands related to Bazaar switch. Running shell commends in Java, esp supporting both Java 1.4 and 5.0, is difficult and brittle.
• Analyzed MySQL runtime data with Weka. Most of the work is in retrieving/preparing the large amount of data efficiently from the database.

KEY TASKS THAT STALLED LAST WEEK

• Still did not collect runtime data with Skoll Client due to Bazaar switch.

KEY CONCERNS

• None

TASKS IN THE UPCOMING WEEK

• Runtime data collection.
• Continue to research tools and methods to analyze runtime information.

Intro

The first working version (0.4.3) of SIGAR I_S plugin is now available on Launchpad. You are welcome to download and give it a try:
bzr branch lp:~m.ch/mysql-server/sigar-plugin
As this product is still in development stage you probably don’t want to install it on you production server. I’d rather suggest to play with it on some non-important MySQL server instance.
So what actually this plugin does. It uses Hyperix SIGAR library to gather and analyze operating system information. As SIGAR library is OS independent we can gather information on different unix platforms as well as on Windows family OS.
What this plugin is not. It’s not a way to start or stop OS services, interrupt its work or affect it in any way.

Features

1. You don’t need to install or compile SIGAR library yourself as the source files come already with SIGAR files and do all the dirty work for you.

KEY ACCOMPLISHMENTS LAST WEEK

• Read the book “Data Mining: Practical Machine Learning Tools and Techniques” to learn about features of Weka.
• Played with Weka to analyze and visualize MySQL runtime data.
• As part of the effort to use push-build and Bazaar with Skoll, found a method to set $PATH in Skoll Client. Now, commands and executables in “non-standard” locations can be specified for Skoll Client (in this case bzr).

KEY TASKS THAT STALLED LAST WEEK

• Could not test or collect runtime data with Skoll Client due to Bazaar switch.

KEY CONCERNS

• None

TASKS IN THE UPCOMING WEEK

• Continue to research tools and methods to analyze runtime information.
• Runtime data collection.