Twelfth week of GSoC

During this week I was mainly engaged in doing some wrapping up work. I started the week with studying about unit testing in PHP. This was totally new to me. I installed PHPUnit into the eclipse environment where I could run the unit tests from the IDE itself. Moreover writing unit tests for PHP was also new to me. I refered to the PHPUnit manual and it was a great source of information to understand how unit testing works with PHP. Also going through the unit tests that was there in PMA I could get some additional knowledge about the writing tests.

With this I started writing unit tests for the code I've written during the summer. I started with unit tests for classes I used for GIS visulaization. Sooner I realized I could make the tests more compact and avoid some code duplications by properly OOPying the tests and went on to do so. I also worte unit tests for the functions in the library files related to GIS.

Later during the week I looked into the documentation of my work. Going through each funcitonlaity I found most of the features to be quite straightforward. However importing ESRI shape files needed some documentation and I provided that as FAQ. I aslo updted the phpMyAdmin wiki with the same details.

With all this my GSoC project has basically come to an end and I'm quite satisfied with the GIS support we have now. However I will work with the team to imporve it further and fix any bugs.

Eleventh week of GSoC

At the beginning of the week I worked on some functionality related to the GIS editor. Earlier I tired to provide the functionality to edit GIS data using the  OpenLayersMaps and it was not a success due to some bugs in the GUI level. As per the instructions of my mentor I revived this effort and the code is in the 'GIS_Editor' branch. I completed the back-end code related to this, but I am still having the same old problems in the GUI level. However the back-end code is working.

Later during the week I looked into the exporting of GIS data into SVG, PDF and PNG formats. I had had a worked on this feature earlier and it was on a working condition. I added the support for labels for these exports and did some small work to make the feature better.

During the next week I'll start the wrapping up work. I will be working on documentation needed, unit test where appropriate, etc, etc.

Tenth week of GSoC

My main task for this week was to implement the 'Function based search' functionality for geometry data. The rationale is that searching for equality or using simple comparison operators is of very less use when it comes to geometry data types. When effectively used with functions defined for geometry types they create powerful search criterion for searching.

I divided these functions mainly into two groups, 'unary functions', i.e. those that take a geometry object as its argument and 'binary functions', i.e. those that two geometry objects as arguments and  test spatial relations between them. On the implementation side, I initially added a 'Function' column for the geometry columns and with that the user is able to perform searched with 'unary functions'. For example the user choosing function ‘Dimention()’, comparison operator ‘<’ and value ‘2’ would add ‘Dimention(`<column_name>`) < 2’ to the where clause.

Later I extended the same concept to binary functions and for those, the geometry object inserted into the value field(in the form of WKT or WKB) is compared with the geometry object in the related column. To make the insertion of geometry objects for comparison easy, I attached the GIS data editor to it, so the insertion is made easy.

During the week I also fixed some of the bugs that arose in the GIS data editor due to multiple row editor being AJAXified. During the next week I will be engaged in my next task, i.e. 'Exporting GIS data into SVG, PDF and Bitmap formats'. A certain amount of work has been done related to this task and I will be completing it during the coming week.

Ninth week of GSoC

Ninth week of GSoC was not productive as I wanted it to be. However I managed to stick to my schedule by the end of the week. 

During this week I worked on finalizing the GIS data editor. I integrated the data visualizing capabilities to the editor, so as the user edits the data visual feedback is provided. Similar to the GIS data visualizer, this provides visualizing is done with SVG and for the data related to earth's geography OpenLayers maps are used with data from OpenStreetMaps. For those browsers that do not support SVG, bitmap images are used. Here the visualization is updated with AJAX while the user is editing.

During the coming week I will be working on the 'function based search' feature which would enable GIS users to perform an advances search using the functions provided in MySQL for GIS data.

Eighth week of GSoC

What I had on schedule for the last week was the GIS data editor. Since I had already started some work on it, basically a version for browsers that does not have JavaScript support enabled, I had a clear idea what I was going to do. During this week I worked on the Ajaxified version of the GIS data editor. The look and feel of it was inspired by the enum editor we have in PMA. 

I worked on the following features.

1) When the users clicks on the pencil icon next to a GIS data field, it asynchronously loads the GIS editor as shown in the image above.

2) When the data type is changed, correct set of input fields are loaded via AJAX and the interfaces is updated.
3) Adding new data points, inner rings, linestrings. polygons are handled with jQuery rather than having to use AJAX

4) When a value is changed the output shown at the bottom is updated.

During the next week I will be working on.

1) Fetching the visualization along with the initial loading of the GIS editor.

2) Updating the visualization when the user changes the data.

As I did previously, I will use both SVG based visualizations and visualizations based on the OpnStreetMaps. For those browsers that do not support SVG (older IE versions), I will use PNG images.

Seventh week of GSoC

This week was spent on completing the feature 'importing GIS data from ESRI shape files'. Last week I managed to implement this feature for ESRI shape types Point, MultiPoint and PolyLine. Extending this to ESRI shape type Polygon was somewhat complicated. MultiPolygon in MySQL was the corresponding type to ESRI Polyogon. However the representation was quite different. ESRI polygons comprise of one or more outer rings and zero or more inner rings. Inner rings are differentiated from outer ones by their clockwise/anticlockwise orientation.  Hence to import them a proper mechanism to categorize each ring as inner or outer, a way to assign each inner rings to its corresponding outer ring were needed. I implemented identification of clockwise/anticlockwise orientation based on the polygon area calculation and implemented a method to determine whether a given point is inside a given polygon. My strategy was to extract a point on the surface from the inner ring and compare it with each outer to identify the correct outer ring. However I hit a dead end when I found that the MySQL function, PointOnSurface()  is not yet implemented. So I had to implement the functionality myself.

Since importing ESRI shape files involves uploading multiple files, last week, I supported this behavior with 'UploadDir' directory where the user has to upload all the relevant files to this directory before importing. Since this is not always available to the user, this week, I enhanced the feature to use ZIP archives for the same purpose. Now the user can ZIP all the files and upload them in one go and the PMA will unpack the archive to 'TempDir' directory and use these files to import data. 

Since I was bit ahead of my schedule I started implementing the GIS data editor -which is my task for the next week- few weeks back. However it turned out to be that this feature will take more time than I originally anticipated. So I will be working on that feature during the next week.

Sixth week of GSoC

During this week I mainly focused on importing data from ESRI shape files. Actually, the task specified in my original proposal for this week was 'exporting GIS data in WKT and WKB formats'. However I felt this is of less value to the end user as it would be trivial for him to use AsText() and AsBinary() functions and then do an export. On the other hand letting the user import data from ESRI shape files, which are heavily used in the GIS arena for a quite a long time, would encourage more people to consider MySQL as a spatial database, as data migration is made easy. Hence with consent from my mentor I started working on this feature.

I used bfShapeFiles library to implement this feature. The author of this library has only implemented functionality to manipulate 2D objects in shape files. This was sufficient for our need since MySQL is anyway supporting 2D shapes only. Though it is known as shape files, referring to .shp extension, shape files consist of 3 mandatory files. .shp files contains the spatial data while a .dbf file which contains the additional data. However to manipulate .dbf files one need to have PHP dBase extension and without this only the spatial data can be imported. Right now I am looking for a parser that can be used to manipulate .dbf files without the dBase extension.

Since the import plugin architecture used in PMA is designed in such a way to support raw files as well as compressed ones I had to extend most of the methods in the library to customize it to our requirement. Since only a single file is allowed to upload in the import process, currently, we need to use the UploadDir to upload all 3 files and then use them if we need to capture data in the .dbf file. However I will be working on using ZIP archives to upload multiple files and then use them in the import process.

Fifth week of GSoC

During this week I started working on the GIS data editor. It is a tedious task to write the Well Known Text(WKT) format of a geometry object, especially when the data type is a complex one and this editor is aimed  at easing the task of inserting new geometry data and editing existing geometry data.

Since I wanted this feature to be available to those who does not have JavaScript enabled, I started with developing a version compatible with JavaScript disabled browsers. Users can reach this editor by clicking the pencil icon next to geometry fields in insert/edit window. The GIS data editor has a dropdown with all the geometry types and choosing a type would change the fields below accordingly. When the user fills the coordinate values and optionally SRID and hit submit, WKT is displayed, so the user can copy it in to the edit window. This is the most basic form of the GIS data editor and I wish to add number of features to enhance the user experience with JavaScript.

Earlier, editing GIS data fields were disabled as browser representation caused problems when submitting changes. Even when the GIS data were not altered they were sometimes captured as altered as described  in bug #3038193 and bug #3194559. While working on the GIS data editor I fixed this by choosing 'GeomFromText' as the default function and having the WKT with SRID in the value field. Meanwhile I had to change the data editing/inserting functionality to accept two parameters (WKT/WKB and SRID) from the value field and add apostrophes correctly.

During the next week I will be Ajaxifying the GIS data editor. By Ajaxifying user will be able to change the data type, add data points, inner rings, polygons without submitting. Further we can provide visual feedback for the changes being done by visualizing the GIS data and to automatically place the final WKT on the value field when the user is done with editing. 

Fourth week of GSoC

Last week I was able to get the 'Visualizing GIS data related to earth's geometry with OpenStreetMaps' functionality up and running. I was mainly concentration on getting the functionality to work rather than on minor details.

This week was mainly spent on perfecting the what was done during the last week. When visualizing GIS data with SVG I used the label column values to provide tooltips. With OpenStreetMaps I could correctly place the label column values on the plot itself. I also worked on proper initial placing and zooming levels. It makes no sense to display the whole world map when the data is only for a particular portion of it. So now the visualizations zoom and pan on initial load to display the data in a more meaningful manner.

I improved the look and feel of the visualization. Now both SVG based and OpenStreetMaps based visualizations are using the same controllers for zooming and panning. Same time I was engaged in improving the documentation comments for the classes and functions I introduced.

More style changes were done to make it look better including the introduction of new icon for the visualization functionality, borders for SVG based visualizations and change of cursor when dragging etc. I also included the OpenStrretMaps JavaScripts in phpMyAdmin which was earlier downloaded on from the respective URL.

Third week of GSoC

During this week I was basically working on visualizing GIS data over a base layer from OpenStreetMaps. Visualizing GIS data this way makes sense only for the data that relates to earth's geography. With each piece of GIS data, MySQL allows you to store a Spatial Reference ID (SRID) which specifies spatial reference systems that relates to the stored data. Hence that need to be taken into consideration when visualizing GIS data.

Going through the documentation of OpenStreetMaps I identified that OpenLayers JavaScript library is an ideal candidate for our purpose. OpenLayers not only support layers from OpanLayers, but it can use data from various sources such as Google Maps, Google Earth, Virtual Earth, Yahoo Maps etc. Further you can add a  vector layer on top of these maps which can then be used to draw standard geometric objects (points, linestrings, polygons). I added Mapnik, Osmarender and CycleMap layers which retrieves up to date data from OpenStreetMaps for display purpose.

I extended the PHP classes I created earlier to support the visualization with OpenStreetMaps. These classes generate the JavaScript code that relates to the geometric objects drawn on the vector layer. I also modified some of the code to take SRID into consideration as it is very important is visualizing data related to earth's geography.

When it comes to using different spatial reference systems, it is important to bring them to a unified reference system for visualization purposes. OpenLayers use proj4js JavaScript library to convert coordinates from one spatial reference system to another. I also used the same and now even the data in the same table which corresponds to different spatial reference systems are brought to a common one (EPSG:900913) for visualization purpose.

During this week I managed to get 'visualizing GIS data with OpenStreetMaps' feature up and running. However a lot more needs to be done to bring it up to the standard. Tooltips, proper initial placing and zooming levels, more options for base layers being few of them. So I will be doing those during the next week.

Second week of GSoC

The main task during this week was to implement the 'Visualization of GIS data' functionality. I had a clear understanding on the functionality and the basic structure for the new class hierarchy as I did some work on the same area to provide a proof of concept prior to GSoC. 

Initially I tried to use flot library for visualizations, for rich functionality it offered, such as zooming and panning, but later realized that using a charting library for this purpose is not suitable. My requirement was to draw basic shapes which charting libraries did not support. So I used SVG to generate the visualizations.

I could use SVG transforms (scale and translate) to support zooming and panning. To manipulate the SVGs I used jQuery SVG, which gave me control over SVG elements. Now double-clicking on the plot zooms it while clicking on the directional arrows on the top right corner pans the plot. Additionally I used jQuery mouse wheel plugin to capture the movement of mousewheel which was then connected to the zooming functionality. Now you can zoom the plot using the mousewheel. Further I used jQuery drag plugin to capture dragging on the plot and now the panning happens when the dragging is performed on the plot. Adding tooltips to visualizations was trivial. So by now I have supplied almost all the rich features flot offered using SVG and a set of jQuery plugins.

One problem with SVG was that versions <= 8 of IE do not support it. Since IE is still the major browser (atleast among non-IT related users) I tried to provide atleast a basic solution to those users. I used GD library to generate the same visualization in PNG format for IE users. 

Since I was looking into generating PNGs I also had a look at 'Exporting visualizations into various formats' feature, even though it was alloted a later time slot in the schedule. Now the visualizations can be exported to SVG, PNG and PDF formats. IMO there needs to be some improvements to this funcionality, but I would like to take them during the allocated time slot. I completed visualizing of GIS data with SVG around a week before its deadline by putting some extra effort, to save some time for the next feature, which might take more time than anticipated. So during the next couple of weeks I will be trying to connect the visualization functionality with OpenStreetMaps to give a better visualization for those data that are related to earth's geometry.

First week of GSoC

Although a full week has not been completed since the official kick off day for coding (May 23rd) it will be much easier for me to write my weekly blog post on weekends. So here I am writing my first weekly report. 

In my opinion the first week was very productive and I could deliver the first two features on schedule. Further I was able to do some more work of the next feature, so most probably I will be able to deliver the next feature before its alloted date.

As the first feature I improved the GIS data browsing by adding two new options. Now the user can view GIS data in standard Well Known Text(WKT) and Well Known Binary(WKB) formats. This is in addition to viewing GIS data with indication of its type and size (eg. [GEOMETRY - 2KB]), which was the method that was there earlier with PMA. 

These formats also adhere to other options set when viewing data. For example when 'partial data' option is set, WKT format will adhere to it by truncating well know text to the specified limit. Similarly WKB adheres to the 'Show binary contents' and 'Show binary contents as HEX' directives.

My second task was to support spatial indexes. Spatial indexes can only be created on spatial data columns and contrary to other indexes, spatial indexes can only have a single column. Further spatial indexes can only be crated with MyISAM and specifying a prefix length is not allowed when creating the index. With this feature, now you can create spatial indexes on geometric columns easily by clicking on the button corresponding to the column in table structure page. 

I also improved the index edit page to suit spatial indexes. To prevent the users from wrongly specifying parameters and ending up with errors, some input boxes are hidden dynamically when 'Spatial' is selected as the index type. Both the above features have been pushed to my repository and you can play around with them on my demo server.

To the end of the week I was working on my next feature which involves a considerable amount of coding compared to the above two. I think this is not the post to write about the next feature. So I will preserve those details to my next weekly post. I am looking forward to deliver my next feature, 'Visualizing GIS data', ASAP as it will always be nice to see some graphics :)

Rigor Automation

When it come software engineering, rigor refers to keeping the code healthy. Code health can be measured in terms of number of parameters, freeness from bugs and vulnerabilities, adherence to coding standards, unit test coverage being few of them. Now, 'rigor automation' refers to the automation of the process of identifying the deviations from this healthy status.

Programmers' lives have been made easy (or one may say hard), with loads of tools that have been built to automate the rigor. Standalone tools as well as plugins to various IDEs, especially eclipse, are there for this cause.

To measure the adherence to the coding style, eclipse plugin, CheckStyle is the most favorite choice. While PMD can also be used for this purpose, it can also be used to detect the potential bugs and vulnerabilities in the code. FindBugs is another great tool that finds bugs as its name suggests. Copy Paste Detector (CPD) does what it’s supposed to do, and it's quite important to find the copy-paste instances as most of the time they are indications of bad OOP design. So you better reconsider your class hierarchy if you have to copy and paste your code in number of places.

Cyclic dependencies between packages are also an indication of bad overall design. JDepend is there to save you by automatically analyzing the dependencies for each and every package to detect the cycles. It's also important to know the statistic about your code. The average, minimum and maximum length of you files, methods etc, the total LOC of you project, comment ratio provides invaluable insights to your code. Tools such as Metrics, SourceMonitor can be used to measure those metrics. Unit test is a must for healthy code and unit test coverage is the measure that provides the assurance in terms of robustness. Coverlicpse and Emma are two great tools that provide unit test coverage statistics.

While most of those listed above are for Java, a number of tools are there for other languages as well. What made me write this post is somewhat related to, me discovering another great tool. While I was looking for a tool that would help me adhere to PEAR coding standards which is used by phpMyAdmin, I found CodeSniffer. Since this post is getting elongated I will write a separate post about my experience with CodeSniffer. 

The Features...

In my previous post I explained the rationale of providing the support for OpenGIS types in phpMyAdmin. In this post I will describe in detail the features I wish to implement under OpenGIS support. 

Following detailed description of features is extracted from my GSoC proposal. I know this is too lengthy for a blog post, but I though this is comprehensive :)

1. Visualizing OpenGIS data

It should be possible for the users to visualize GIS data wherever applicable. When the GIS data refer to the Earth’s geography (i.e. when Spatial Reference ID is present) visualizing GIS data is done using the API of OpenStreetMap. On other occasions a simple visualization is presented. Zooming and panning is provided  for better user experience.

When the user browses a table or views a query result that has atleast one OpenGIS data column a link should appear underneath the table to visualize the GIS data. User can chose the GIS column to be visualized if more than one GIS column is present and chose a label column that is used to generate tooltips and/or legend.

Here is the proposed interface layout for the visualization.

 2. Inserting OpenGIS data

It is a tedious task to write the Well Known Text (WKT) format of a geometry object, especially when the data type is a complex one. This difficulty is addressed by automatically constructing the WKT from the points the user provides.

When the GIS data refer to the Earth’s geography, points for complex geometry types are captured by tagging on OpenStreetMap maps. When the GIS data refer to simple geometry WKT is constructed by X and Y cordinates the user feed in for each point. The user is provided with visual feedback of the geometry object being created and user can click on the visualization to capture the clicked location and use it as a point in the OpenGIS data object.

Here is the proposed interface to insert multipolygon data objects. The interface changes appropriately for each of the OpenGIS data types.

 3. Editing OpenGIS data

Currently editing OpenGIS data fields is disabled as browser representation causes problems when submitting changes. Even when the GIS data is not altered they are sometimes captured as altered. I wish to address this problem before developing the new interface to insert/edit GIS data.

A similar interface used to insert data is used to edit OpenGIS data. User can modify the data by adding new data points and/or by deleting some of the existing data points.

4. 'Function based search'

Searching for equality or using simple comparison operators is of very less use when it comes to OpenGIS data types. When effectively used with functions defined for OpenGIS types they create powerful search criterion for searching. I wish to implement 'function based search' for OpenGIS data types to enhance the search capability under table search.

       a.   Unary operator functions

The user will select the function, the comparison operator and enter the comparison value. For example the user choosing function ‘Dimention()’, comparison operator ‘<’ and value ‘2’ would add ‘Dimention(`<column_name>`) < 2’ to the where clause.

Another extension to 'function based search' is function chaining. For example applying ‘Envelope()’ function to a geometric object would yield a polygon and we can apply 'function based search' to the resultant polygon. Facilitating this would require a highly dynamic interface and I wish to support this idea as an optional deliverable. However I would do my best to deliver this within the stipulated GSoC development period.

       b.   Binary operator functions

Search based on special relations between geometric objects is essential for an effective GIS search functionality. For example user would want to search for all geometric objects that lie inside a user given polygon or to search for all geometric object within which the given polygon lies. Moreover search functionality need to be capable of searching based on whether geometric objects cross, intersect, overlap, are disjoint etc. Similar search capabilities would be sought with related to the Minimum Bounding Rectangle (MBR) of geometric objects.

5. Exporting OpenGIS data

When exporting either database or table the user should be asked to indicate whether the Geometric data should be exported in WKT or WKB formats. In the case of exporting to SQL, compatibility should be ensured so that the data can be imported back without lose of data or data types.

Exporting GIS data in visual form makes sense only for a single geometric data column. I would like to extend this capability to bitmap images and PDFs using GD extension and TCPDF library respectively.

6. Spatial indices

When the user adds an index to an OpenGIS data column, by default a SPATIAL index should be added.

7. Browsing OpenGIS data

Three options would be added in ‘Options’ section of query result browsing window

To view geometric data in Well Known Text (WKT) format

To view geometric data in Well Known Binary (WKB) format

To view geometric data with Geometric type indication and size (e.g. [GEOMETRY – 2.2KiB])

Rationalé

OpenGIS data types are supported in MySQL since version 4.1. However the users of OpenGIS data types are deprived of numerous functionalities that are being offered by phpMyAdmin on other data types.

OpenGIS data types are different to most of the other data types supported by MySQL. For people to make sense of OpenGIS data it is necessary that the data is visualized. This project addresses this issue by visualizing GIS data in numerous instances.

Typing Well Known Text format to enter or edit GIS data is a pain when the data being entered is complex. This project will make it easy for the users to enter GIS data, both simple data types as well as those relate to a spatial reference ID (SRID), using intuitive interfaces in both the cases. A similar improvement is proposed for editing existing data.

The search functionality offered to other data types is not sufficient for GIS data. For example a user might be more interested in searching for all the points that lie in a user specified polygon rather than searching for a point based on equality or a simple comparison criteria. I wish to facilitate this requirement by extending the traditional search with ‘function based search for OpenGIS data types’.

My GSoC Project : OpenGIS support for phpMyAdmin

I'm overwhelmed that my proposal to implement OpenGIS support for phpMyAdmin under GSoC 2011 got accepted. A big thank goes to phpMyAdmin team and Google for this great opportunity.

I will use this blog to introduce you to the project as well as to keep all of you updated about the current status of the project. So the next few posts will rationalize the need of OpenGIS support for phpMyAdmin and introduce  the features I am going to implement under the project.