Monday, 20 May 2013

Mercury: The WYSIWYG html editor

I had this application where different users would want to edit custom html pages to be shown up in there web sites. Each user will have his own domain( all domains pointing to the same Rails application ) and the custom html page had to be loaded as per the current domain. To do this, in search of a WYSIWYG html editor which is easy to setup and simple to start off, I ended up in Mercury. Whats really nice was that Mercury also had a gem to be used for Rails developer and as I am one, I had no more hesitation in get started with mercury.

To get started off with mercury, add

[code language="ruby"]

gem 'mercury-rails'

[/code]

to the Gemfile and bundle it.

Run the rails generator for the mercury files.

[code language="ruby"]

rails g mercury:install

[/code]

A couple of questions will be posted. Press 'yes' to install the layout files.

Now checking out the directory structure,  you could see three additional files.

mercury.js and mercury.css in the js and stylesheets assets respectively. Also, a new layout file for the mercury editor, mercury.html.erb 

I did remove the mercury css file later on.

One thing that needs to be noticed here is that the mercury.js file is heavy and it woudn't be a good idea to load it in all the pages. We would want to load it in only the pages that needs to be edited. Checkout the mercury layout file and you can see that the mercury.js file is included.

[code language="ruby"]
<head>
<meta name="viewport" content="width=device-width, maximum-scale=1.0, initial-scale=1.0">
<%= csrf_meta_tags %>
<title>Mercury Editor</title>
<%= stylesheet_link_tag 'mercury' %>
<%= javascript_include_tag 'jquery-1.7', 'mercury' %>
</head>
[/code]

Now to prevent mercury.js from being loaded up in the pages, we could move all the other js files in our application to a separete directory and then require the directory in our application.js

My application.js will have,

[code language="ruby"]
//= require_tree ./main
[/code]

where main is the directory which has all the application specific javascript. (Probably could be a better name :) )

Now peep into the routes file, you could see this extra line,

[code language="ruby"]

mount Mercury::Engine => '/'

[/code]

What this line does is that it allows the html pages in your application to be edited. An extra '/editor'  will have to be added at the beginning of each url path to load the mercury editor for the page.

Consider you have the url 'localhost:3000/pages' , all you need to load it in the mercury layout is to change it to ''localhost:3000/editor/pages' . You have mercury loaded up to edit your page and can now see it in the mercury editor's layout.

Screenshot

However this isn't just enough to start editing the page. You need to specify editable regions in the page.
In pages.html.erb 

[code language="ruby"]
<div class="control-group">
<h3 class="section_header page-header">Pricing page</h3>
<div id="faq" class="mercury-region" data-type="editable" data-mercury="full">
<%= render :file => file_path(@domain 'faq') %>
</div>
</div>
[/code]

Consider this piece of code. A div with id="faq" is made editable with class="mercury-region" and attributes data-type="editable" and data-mercury="full".

Now you can see the editable region.

Screenshot-1

This following line in above piece of code

[code language="ruby"]
<%= render :file => file_path(@domain, 'faq') %>
[/code]

invokes a helper method and loads the already created sample faq template which can now be edited and saved for the particular domain. As simple as that.

Similarly you could edit more pages here. This is how the contacts page can be edited.

[code language="ruby"]
<div class="control-group">
<h3 class="section_header page-header">Contact page</h3>
<div id="contact" class="mercury-region" data-type="editable" data-mercury="full">
<%= render :file => file_path(@domain, 'contact') %>
</div>
</div>
[/code]

Also, you probably might want to change the save url of the mercury editor for the particular page. That is the controller action to which the mercury edited contents will be 'POST' or 'PUT' (depends on the configuration set in the mercury.html.erb)

To change the mercury save url for this particular page, I wrote the script in the erb file ( pages.html.erb )

[code language="ruby"]
<script>
$(window).on('mercury:ready', function () {
Mercury.saveUrl = "<%= pages_upload_admin_domain_path(@domain) %>";
});
</script>
[/code]

You might also want to change the page thats to be redirected to once we are done with editing using mercury. We could bind on mercury's save event to get this done.

[code language="ruby"]
$(window).bind('mercury:saved', function() {
$(window.location.replace('/admin/domain'));
});
[/code]

All this saved data would have to be dealt with in the controller action. Inspecting the params in the controller action ( the mercury Save url) ,
{"content"=>
{"faq"=>
{"type"=>"full",
"data"=>{},
"value"=> "<h1>This is where I have done my FAQ editing</h1>"
"snippets" => {}
}
},

{"contact"=>
{"type"=>"full",
"data"=>{},
"value"=> "<h1>This is where I have done my Contacts editing</h1>"
"snippets" => {}
}
}
}


There are two things of notice here. The contents hash contains all the mercury related stuff.  Each hash in the contents hash has a key which is equal to the id of the mercury editable html divisions ( see the view code pasted above ), here 'faq' and 'contact'. The actual edited html content can be found in the hash with key 'value' ( <h1>This is where I have done my Contacts editing</h1>).  'The controller action could decide on how to save this html content.

What have I done to solve my case mentioned at the starting?

I created a pages directory in my public. Within the pages directory I created sub directories which corresponds to the domain. For eg, the domain localhost corresponds to the directory named localhost inside the public/pages directory and the domain remotehost corresponds to the remotehost directory.

I then saved all these edited html content as html files within these domain specific directories. When a particular domain was loaded, the html pages ( for eg, faq and contact) was rendered from the corresponding domain directories in the public folder .

Sunday, 19 May 2013

Delayed Jobs in Rails: Adding custom attributes

Ok, so this was my exact scenario. When I was doing a bulk emailing application,  there was the need for the client to upload his set of email ids as a file and then save it to the database. The process of saving these contact mail_ids for a particular mail group was a delayed process, handled by Rails delayed job . 


[code language="ruby"]
@mail_group.delay.save_group_contacts
[/code]

where @mail_group is the active record group to which the mails_ids being uploaded and saved belong.

The requirement was to show a progress bar for the process of the mail_ids being saved to the the mail group. To handle this, I decided to add custom attributes to the delayed jobs table so as to identify the owner of the delayed job and also find the progress of the job.

To do this,

1) DB migration to add the custom attributes

[code language="ruby"]
class AddColumnToDelayedJob < ActiveRecord::Migration
def change
add_column :delayed_jobs, :job_process_status, :integer, :default => 0
add_column :delayed_jobs, :job_owner_id, :integer
add_column :delayed_jobs, :job_owner_type, :string
end
end
[/code]

2) A model for the delayed jobs table.

[code language="ruby"]
module Delayed
class Job < ActiveRecord::Base
self.table_name = "delayed_jobs"
attr_accessible :job_owner_id, :job_process_status, :job_owner_type
belongs_to :job_owner, :polymorphic => true
end
end
[/code]

As seen, three extra attributes (job_owner_id, job_owner_type attributes for establishing a polymorphic association with the job owner of the delayed job and a job_process_status attribute for updating the progress of the job) were added to the delayed jobs table.

Delayed jobs were then created with the job_owner_id and job_owner_type.

[code language="ruby"] @mail_group.delay(job_owner_id: @mail_group.id, job_owner_type: @mail_group.class.name).save_group_contacts[/code]

However this would not be enough to update the custom attributes. An attempt to create a delayed job would produce this

[code language="ruby"]
ActiveModel::MassAssignmentSecurity::Error:
Can't mass-assign protected attributes: job_owner_id, job_owner_type
[/code]

As a quick fix, add a config/initializers/delayed_job.rb
and paste in the following code

[code language="ruby"]
class Delayed::Job < ActiveRecord::Base
self.attr_protected if self.to_s == 'Delayed::Backend::ActiveRecord::Job'   #loads protected attributes for  # ActiveRecord instance
end
[/code]

Now the delayed job would get saved with the job_owner_id and job_owner_type.

Also, in the mail_group model, set an association to the delayed jobs table.

[code language="ruby"]
class MailGroup < ActiveRecord::Base
has_many :deferred_jobs, :as => :job_owner, :class_name => "::Delayed::Job"
end
[/code]

Now you can access all the delayed jobs of a particular @mail_group as

[code language="ruby"] @mail_group.deferred_jobs[/code]

The job process status which is updated by the running job can also be accessed as

[code language="ruby"] @mail_group.deferred_jobs.job_process_status[/code]

Sunday, 18 December 2011

Git Reset, Revert, Merge Conflicts and the Case Of The Faulty Merge

Git, as we know is a fast, open source, distributed version control system that is quickly replacing subversion in open source and corporate programming communities. As a developer, many a times i have been amazed by the power of git and how it takes care of our code repo. It track files in the project, we periodically commit the state of the project when we want a saved point. The history of our project is shared with other developers for collaboration, merge between their work and ours, and compare or revert to previous versions of the project or individual files.

As mentioned earlier, Git, at a fast pace, is replacing subversion in open source and corporate programming communities. Hence most open source developers would have had a taste of git and its power. We all would have done a git init, push, pull, rebase and stuff in our day to day programming activity and those would be quite trivial to most developers.

However there are certain facets of git(merges, conflicts, reverts and such) which does create some kind of confusion to developers, at least when they use it for the first time. What made me write down this post is an incident that happened to my colleague while he was on work. Will get into that shortly.  Before  getting into that, let me just stitch in a brief on Revert and Reset in git.

Revert and Reset

Git provides us multiple methods for fixing up mistakes while in development mode. This is important, because it saves not just our work but the others who are involved in the same project.

If you have actually done a mess with your working directory, but actually haven't committed the changes, the best way is to perhaps do a hard reset.

$ git reset --hard HEAD


This would just wipe off the changes that you have made in your git index and also any outstanding changes that you have made in your repo.

Now suppose you have committed your changes, but haven't pushed it into master,  and then suddenly you feel like you shoudn't have made the previous commit(or a sequence of your previous commits), you could again reset hard. This is as simple as doing

$ git reset --hard HEAD~n


This would set the HEAD of the git index to 'n' commits prior to your current head. The problem though with doing a git reset --hard is very obvious. This is how your commit log looks like with at A its head

o  ->  o  ->  o  ->  D  ->  C  ->  B  ->  A


Suppose you do

$ git reset --hard HEAD~3


Now your commit log would be.

o  ->  o  ->  o  ->  D


This means that the changes that you made right from A to C have been vanished and you are not going to get it back. The bottom line is simple. You are not able to change the effects made by a single commit(ofcourse, the exception is your last commit as we have already seen).

git-revert is just for that.

The current commit log would look like this

o  ->  o  ->  o  ->  D  ->  C  ->  B  ->  A


At any point of time, you realize that 'C' is bound to break your code(hopefully it still hasn't), you may well want to undo the changes made by C. This could be done by

$ git revert (commit id of C).


This would create a new commit that undoes the commit C. You will be given a chance to enter a new commit message, but the default message that indicates its 'the reverse of the commit C' would be the most indicative commit message to have.

o  ->  o  ->  o  ->  D  ->  C  ->  B  ->  A  ->  rC


where rC is the reverse of C.

This revert is a straightforward revert(i.e. it just undoes the data made by the commit reverted). Since all thats being talked about is a single branch, there aren't any complications that would arise here.

Merge and reverting a faulty merge

Now let me talk about the incident that i had mentioned earlier. All these happened as a result of an accidental Merge. My friend did this

$ git pull origin experimental


while he was still sitting in his master branch. The experimental branch has now been merged into the branch master. This was totally unintentional(he never planned to do a merge). There were no merge conflicts however. The mainline code broke. We had to revert this faulty merge.
Master  ->            P   ->   x    ->  M
                                \                     /
                                   \                /
Experimental ->        A    ->   B

This would give you a picture. P is the point of branching. x is some commit made in the mainline branch totally unrelated to the side line branch. The side line branch itself has got two commits of its own, A and B. M is the merge commit (experimental has been merged with master). The code broke. Hence, we need to revert M(the merge commit).
Master  ->            P   ->   x    ->  M  -> W
                                \                     /
                                   \                /
Experimental ->        A    ->   B

Now as seen, the merge has been reverted(W is the reverse of M). This was done with

$ git revert -m 1 (Commit id of M)


This adds W to the commit log as well. Now the faulty code in the experimental branch was worked upon and fixed and its been made ready for the merge (again!). The experimental branch is now merged with the master branch. What was weird(for us, at that point of time) and noticeable was that the code changes that were made after the 'merge revert' appeared in the master branch whereas the ones made before the revert didn't appear. i.e.

Master - >          P -> x -> M -> W -> x -> x -> M2

Experimental ->        A -> B  -  -  -  -  -  -  -   C -> D


Again, x are the commits unrelated to the experimental branch. M2 is the second merge. Commits in the experimental branch,C and D, fixes the faulty code in A and B. Whats to be noticed is that, after the updated experimental branch has been merged, none of the changes made by A and B would appear in the master branch, whereas the changes made in C and D would.The reason was found out soon.

Linus Torvalds explains the situation:

     Reverting a regular commit just effectively undoes what that commit
     did, and is fairly straightforward. But reverting a merge commit also
     undoes the _data_ that the commit changed, but it does absolutely
     nothing to the effects on _history_ that the merge had.

     So the merge will still exist, and it will still be seen as joining
     the two branches together, and future merges will see that merge as
     the last shared state - and the revert that reverted the merge brought
     in will not affect that at all.

Thats what just happened here. W(merge revert) undoes the data made by M(merge) but does nothing to the commit history brought in by M.There fore when the second merge,M2, is made, the commit history is checked and M is found to be 'last shared state'. Hence, only those changes that has been made after the 'last shared state', M, will be merged into the master branch now(i.e. commits C and D). None of the data created in A and B would merge, because as per the commit history, they are already merged.

Solution to this problem is also explained by Linus himself. The fix is to 'revert the revert that brought in W', i.e, revert W before you do in the next merge,M2.

Thus the main line commit log would be

P  ->  x  ->  M  ->  W  ->  x  ->  x  ->  Y  ->  M2.


where Y is the reverse of W and M2 is the merge made after that.

$ git revert (commit id of W)


adds Y to the commit log. The above commit log would be equivalent to

P  ->  x  -> M  ->  x  ->  x  ->  M2


where there is no W nor a Y and then the second merge has been performed, M2. Now this would be fine, and all the changes made in the experimental branch should be seen in the master branch(ignoring merge conflicts). If there are any merge conflicts arising, git leaves the index and the working tree in a special state that gives us all the information needed to resolve the merge.

Merge Conflict

A Merge conflict would throw in the following message:
CONFLICT (content): Merge conflict in sample_script.rb Automatic merge failed; fix conflicts and then commit the result

Trying to switch to the experimental branch would give you this
error: you need to resolve your current index first

The files with conflicts will have markers upon them.
<<<<<<< HEAD:sample_script.rb "We would be starting off now" ======= "This would be the end" >>>>>>> d31f96832d54c2702914d4f605c1d641511fef13:sample_script.rb

Now we need to resolve these conflicts manually followed by adding the file and commit it.

$ git add sample_script.rb
$ git commit -a


The commit message would already be filled in indicating that its a conflict resolving commit. I always prefer not to add in anything extra on that.

gitk

It would also be helpful to have the 'gitk' tool when you are analyzing your commit logs, specially when you have more than once branch. You would be given a neat graphical representation of your working directory.

$ sudo apt-get install gitk


if you already don't have one.

Image

This definitely would be helpful in getting a better picture.

Tuesday, 12 July 2011

Solution to 'Wireless disabled by Hardware switch'

Ohhh ... Suddenly my wifi goes off( a combination of DELL with Ubuntu natty) and got a display - wireless disabled by hardware switch. Got down to googling and finally got this.

To get back your wifi to start working, type in the following:

$:  sudo rfkill list

You will get back something like this.

0: dell-wifi: Wireless LAN
    Soft blocked: yes
    Hard blocked: no
1: dell-bluetooth: Bluetooth
    Soft blocked: no
    Hard blocked: no
2: phy0: Wireless LAN
    Soft blocked: yes
    Hard blocked: no
4: hci0: Bluetooth
    Soft blocked: no
    Hard blocked: no

As you could see (0,2) the Wireless Lan has been soft blocked.  We need to unblock it to get it up and working properly.

$: sudo rfkill unblock 0

$: sudo rfkill unblock 2

 

Restart your networking

$: sudo /etc/init.d/networking restart

 

and yup! the wifi should be back now.

Monday, 4 July 2011

RubyconfIndia 2011

May 27,28 - Attended Rubyconf India 2011 held at Royal Orchid hotel, Bangalore.'  For some one who has less than a year of hands on with ruby, it was great hearing from the giants - the very Matz himself expressed his love for the community(in his own peculiar Japaneese way), Ola bini, Chad Fowler,  Brian and others. What in short ? - awesome 2 days.









Module functions as class and instance methods

Consider you have a module and a class.

module Mymod and a class Myclass.

The situation in hand is such that certain functions in the module need to end up being instance methods of the class Myclass and certain functions need to be Class methods.  You could very well imagine of such situations. Consider you are using ActiveRecord and have a sub class Subscription in correspondence with a DB table. You want to insert logic, within the module, that would work in each of the following case.

1) When a subscription fails or succeeds.

2) When an unsubscription fails or succeeds

You do this.

module SubscriptionLogic


   def  after_sub


      ....


   end


   def  after_unsub


     ....


   end


   def  after_sub_fail


     ....


   end


   def  after_unsub_fail


      ....


    end


end


class Subscription

   include SubscriptionLogic

   .....

end

You insert the logic as functions of a module, say SubscriptionLogic. Ideally you want the methods containing the logic to be instance methods of the class Subscription. You include the module SubscriptionLogic in the class and you avail all the functions in the module as Instance methods. Now you consider the case when an unsubscription fails - i.e, there is no existing subscription so that an unsub could take place.  No way is it possible that you could have a function 'logic_after_unsub_fail'  in the module SubscriptionLogic and use it as an instance method, simply because there is no instance available.  You think and decide to use the function as your class method, but you have 'included'  the  module in your class and hence its not possible to use it as your class method. You cannot extend the entire module coz , ideally you want the logic to be instance methods.

So you could get this solved up by a simple piece of extra coding.

module SubscriptionLogic

  def  after_sub

      ....

  end

  def  after_unsub

      ....

  end

  def  after_sub_fail

      ....

  end

  module ClassMethods

      def after_unsub_fail

           ....

      end

      def self.included(base)

          base.extend(ClassMethods)    # base pertains to the class within which you include the module.

      end

   end

end

Now within the class insert this line.

class Subscription

   include SubscriptionLogic

   extend SubscriptionLogic::ClassMethods

   .....

end

Sunday, 3 July 2011

Back after a Long Gap...

Haven't Written any thing for  quite some time now. It was last november that i actually sat down to write something. Started off my professional career then and obviously was busy with work and stuff. Need to get back to writing stuff, little and bigger stuffs that i have learnt while working. In between i have fallen in love with a a new language - Ruby, the popular choice for web apps these days.  From a person who never had any kind of fondness for OOP, i have started loving to talk about about objects and classes. Learnt the difference between fun coding and professional coding. These 6-8 months has really been a learning curve for me and i really do hope that my following posts would reflect that.

Saturday, 27 November 2010

Getting started with PHP and apache

Someone who gives it a thought of using PHP , apache and stuff for the first time will be faced up with a few number of fixed problems . Configuring your apache server , establishing a connection between PHP and apache , stuffs like that . Here is a basic introduction to start using these things and kick off as a web programmer .

Obviously you need to have PHP and apache installed on your system .

$ : sudo apt-get install apache2

$ : sudo apt-get install php5-cli

$: sudo apt-get install libapache2-mod-php5

 

would get you apache and PHP installed on your system  .

Using the appropriate versions is left upto individuals .

Just have a peep into some of the files like :

/etc/php5/apache2/php.ini -----> You would see a lot of variables set to 'ON'  and 'OFF' . You may well have to toggle those values as and when necessary .

 

Try starting , stopping and restarting your apache web server .

$ :sudo  /etc/init.d/apache2 start

$: sudo /etc/init.d/apache2 stop

$: sudo /etc/init.d/apache2  restart

 

Once everything is fine and and perfect  , try out your first PHP script . Script it , save it in a file with a '.php' extension and put your file in the directory '/var/www '. Now you could access this script from your browser

localhost/'scriptname'

and you see the result right there in front of you . Thats just how you enter the world of web programming .

Wednesday, 24 November 2010

GreaseMonkey : Customizing your webpages

There would have been times when you would have felt if the web page in front of you behaved a bit more according to your like . This might specially happen if the page consists of forms that you fill with data directed to live servers .  In such cases you wouldn't mind customizing the page to suit your needs so that you don't do some nonsense that would result in chaos .

Firefox , i see many using it as their default browser , offers a plugin called 'Grease monkey'  that allows you to do just that . All you need to is to install the plugin , write your own javascripts , install them ( thats pretty easy to do ) and start using them  .

Write your javascripts to customize your webpage . What you need to do in addition is to add some metadata providing extra information to greasemonkey .

The meta data might look as follows :



 

 

 

 

 

 

The first and the last lines are necessary to indicate to the greasemonkey that the script it to be processed by it .

The name and the description are optional . It might prove helpful to you if you have scripted and installed a number of greasemonkey scripts .

The 4th an the 5th line of the metadata are extremely important and needs to be well understood . These lines tell the greasemonkey the URLs on which the script is to be applied and not to be applied .

@include -> specifies the URLs on which the script is to be installed . Here '*' is specified which is just a wildcard and indicates that the script is to applied on all the sites  .

@exclude -> specifies the URLs that are to be left out . It is to be noted that @exclude is processed before @include .

This gives you an idea of the metadata part of the script . Now you may write your own javascript , add your metadata at the beginning and save the file with the extension ' .user.js '  .

To install the script , open the file in firefox and you would see an option 'install' . Just give a click and you are ready to use your greasemonkey script .You could manage your script , disable it , uninstall it as well . Just click

Tools ----> Greasemonkey ----> Manage user scripts ....

Having another plugin 'firebug' may help you in the process of writing the script . This allows you to view the html code of the webpage that is infront of you . Infact the plugin is almost a necessity to help you write efficient code with greasemonkey .

Now you may start writing your own greasemonkey scripts and customize any of the webpages as and according to your like . It's also to be understood that greasemonkey is a plugin of firefox . If you use Google chrome as your browser , you wouldn't need to install any plugin . The facility to customize the webpage is provided by default . You may use the same code that you have written to work with greasemonkey . The metadata and the process of installing is the same and obviously the results are the same as well .

Using Screens

You might have often had the experience where you would have had to open quite a few terminals on your screen simultaneously . This might often be the case when you are doing stuffs like data processing on particular files and you intend not to sit idle during that particular time . What you do is to keep unnecessary number of terminals open on your screen .

You could get rid of this by using the 'screens' option in Linux . This allows you to enter a new screen where you could perform time consuming processes like a data processing operation using the  'awk'  command . Start the process in the screen , exit the screen and now do your own stuff in the main console , go back to your screen after a while and view the results of  whatever operations you had performed . You could do all this stuff without the necessity of opening terminals again and again .

Create a new Screen :

$ : screen -S 'Screen name'

To detach from the current screen :

Use CTRL + a + d

To terminate a screen :

Use CTRL + d

To attach an already created screen :

$ : screen -x 'screen name'

You could keep on creating screens within screens and keep going on as and when needed  .

Tuesday, 26 October 2010

Semaphores and Synchronization Patterns

Semaphores as we know is a mechanism that is used to solve the issue of synchronization . The concept and use of semaphores is a bit more than what you normally call ' tricky ' .Here we try and analyze a basic synchronization pattern and solve the issue with semaphores .

Have a look into the pattern given below .


statements in THREAD A


strcpy(a , "Hello ");   -  A1
printf("%s\n" ,b);      -    B2


statements in THREAD B


strcpy(b , "World")  - B1
printf("%s\n" ,a);    -  A2


Here we have two threads which consists of two statements each .
Constraint :
What we need to ensure is that the statements suffixed A1 & B1 are executed before the statements suffixed A2 & B2  execute. Here the two threads 'rendezvous ' at a point of execution and is not allowed to proceed until both have arrived .

Without using a mechanism like semaphore , no way is it possible to ensure such a constraint due to the scheduling mechanism that cannot be predicted . Hence we use semaphores to solve this issue .

To ensure that it works according to wish  , we do this :

THREAD A


strcpy(a , "Hello ");  - A1
up(sem1);
down(sem2);
printf("%s\n" ,b);   - B2


THREAD B


strcpy(b , "World");  - B1
up(sem2);
down(sem1);
printf("%s\n" ,a);   - A1


Now we need to give a thought on how this works .
Here we have two semaphores ,sem1 and sem2 as we have two critical sections

Now , how is the synchronization issue solved .?
The 'down ' operation just before the two statements suffixed '2' ensures that the two statements are not executed until the 'up' operation on the corresponding semaphore is not invoked . Hence whichever be the order of scheduling , both the threads rendezvous at a point and will not proceed ( as a result of the 'down' operation )until the next thread arrives (and the corresponding 'up' operation is performed ).

This synchronization pattern as been solved using semaphores .

You could find the solution to these and more such patterns from here .

Wednesday, 20 October 2010

Memcheck : A Memory Error Detection Tool

Linux distros provide a  tool suite called 'Valgrind' that consists of a number of tools that help you make your programs faster and more correct . The most popular tool out of this is called is memcheck that is used to detect memory related errors in C and C++ programs that can lead to serious issues like  segfaults and even more grevious issues like unpredictable behaviours .

Lets take a look into how we use the tool 'memcheck' so that we detect , try and avoid the memory related issues in our program.

First of all we need to create a memory leak so that we have something to detect .

We write a program that consists of a memory leak . Here is one such in a file ' pgm.c'

1 #include <stdlib.h>
2 #include <string.h>
3 void f(char *s)
4 {
5 char *x = malloc(strlen(s));
6 x[7] = 0;
7 }


8 int  main()
9 {
10 char *s="Hello";
11 f(s);
12 }


As you could see , there are two major memory related bugs in this code:

1) Trying to write in the location x[7] which is out of the allocated memory space .This will lead to what is called a 'heap block overrun' .

2) The memory that is allocated to x remains unused . This memory becomes garbage on returning from the function . In short , you have seen a 'memory leak '.

Now lets try and detect the two problems using memcheck :

You need to have valgrind installed in your system .
$ : apt-get install valgrind
if you don't have it already .

Do

$: cc -g pgm.c

We  compile the program using the -g option so that the line number informations are displayed when using memcheck .

Do

$: valgrind  --leak-check = yes a.out

the option  ' --leak-check ' being set equal to yes displays the informations on memory leak issues .

Now lets look out what are the informations that are being displayed when the memcheck tool is used .

First lets have a look into the 'heap block overrun' problem :

==3350== Invalid write of size 1
==3350==    at 0x80483F6: f (pgm.c:6)
==3350==    by 0x804841D: main (pgm.c:11)
==3350==  Address 0x419102f is 2 bytes after a block of size 5 alloc'd
==3350==    at 0x4023D6E: malloc (vg_replace_malloc.c:207)
==3350==    by 0x80483EC: f (pgm.c:5)
==3350==    by 0x804841D: main (pgm.c:11)


The above few lines is the code that was generated by memcheck  .

'3350' is the process id .

The actual error is seen right at the first line .

'Invalid write  of size 1' .

You get a stack trace right after this line , that

the invalid write has occured at the 6th line as a result of the 12th line . You could see what line numbers '6' and '11' do by having a look into our code , its the point of occurence of the error and the function call respectively.

A line showing the the fact that the location you are trying to access ( x[7] ) is 2 bytes after the allocated area can also be seen added with lines that contain information about the main and the function .These lines provide great help to the programmer especially when the case becomes a lot more complicated .

Now the informations that are displayed about the 'memory leak problem' can also be looked into .

==3350== LEAK SUMMARY:
==3350== definitely lost: 5 bytes in 1 blocks.
==3350== possibly lost: 0 bytes in 0 blocks.
==3350== still reachable: 0 bytes in 0 blocks.
==3350== suppressed: 0 bytes in 0 blocks.

You could view the lines that provide information about the memory leak problem .
The first line of the 'LEAK SUMMARY ' is the most significant to us . It shows the amount of memory definitely lost ( 5 Bytes ). Changes need to be made in the program so that the memory leak is prevented .

Memcheck produces these result which helps the programmer so as to view and correct the memory related issues rather convincingly . It needs to be noted that Memcheck is a 'dynamic instrumentation tool ' and not a static tool like 'lint ' . Hence to detect the memory leaks in a program , the control actually needs to get transferred to that segment of the program where the issues occur . In short you need to invoke the function 'f' from your 'main' so that Memcheck could detect those memory related issues that exists within the function 'f' .

Tuesday, 19 October 2010

Common Subexpression elimination ( CSE) by the GCC Compiler

Common subexpression elimination (CSE) is a compiler optimization technique of finding redundant expression evaluations, and replacing them with a single computation . This saves the time overhead resulted by evaluating the expression for more than once . We will have a look into this phenomenon by considering a simple code and again taking a walk through its assembly code .

Consider the code that we have written and saved in file ' pgm.c '

main(){
int i, j, k, r;
scanf("%d%d", &i, &j);
k = i + j + 10;
r = i + j + 30;
printf("%d %d\n", k, r);
}


Do :

$ : cc -S pgm.c

$ : less pgm.s

Read the assembly code ' pgm.s '

Scan the assembly code and find out the call to the function 'scanf()' . This is where our area of interest begins . This is because its after calling 'scanf ' that we load the value of the input variables into the registers .

Right after the call to 'scanf ' , i have this in my assembly code ,

movl    -16(%ebp), %edx
movl    -20(%ebp), %eax
.

Here we move the values of the two variables 'i ' and 'j ' into registers 'edx ' and 'eax ' .

Now just have a look into our C code and you will see the two expressions as follows :

k = i + j + 10;
r = i + j + 30;


Looking at the two expressions , you would find out the redundant part in the two expressions .

Its that 'i+j' is calculated twice .

Now lets find out what happens in the assembly code :

I had these statements in your assembly code :

movl    -16(%ebp), %edx

movl    -20(%ebp), %eax ----------> We had mentioned these two statements  above .

leal   (%edx,%eax), %eax -------->  values in edx and eax are added and loaded into eax

addl    $10, %eax -------------------->  the constant 10 is added with the value in eax

leal   (%edx,%eax), %eax --------->   values in edx and eax are added and loaded into eax

addl    $30, %eax -------------------->  the constant 30 is added with the value in eax

You could see that there is redundancy in the assembly code .

The statement ' leal(%edx,%eax), %eax ' performs the evaluation of the subexpression ( ' i + j ' ) which is done twice thereby throwing in redundancy and hence creating unnecessary overheads .

Obviously our compiler when asked to perform optimization would avoid this redundancy .This is what is termed as Common Subexpression Elimination ,i.e , the subexpression that is common( ' i + j ' ) in the two expressions is evaluated only once , in essence the second evaluation is eliminated .  Lets have a look into CSE by again peeping into our assembly code  .

Do

$ : cc - S - O3 pgm .c

$ : less pgm.s

Take a look into your assembly code  .

As mentioned above , our area of interest starts from the call to 'scanf ()' .

Instead of those 6 statements that were used to evaluate those two expressions ,

i + j + 10 & i + j + 30 ,

you would find this ,

movl    -12(%ebp), %eax ---------------->  move the value of the variable 'i' into eax

addl    -8(%ebp), %eax -------------------> add the value of the variable 'j' with 'i' stored in eax .

leal    30(%eax), %edx -------------------> adds 30 with eax ( contains the sum of i & j ) and stores in edx

addl    $10, %eax ---------------------------> adds 10 with eax ( contains the sum of i & j ) and stores in eax .

Here the second statement performs the evaluation of the subexpression , ' i + j ' . You could get it quite clearly from the code segment that the redundancy that was found in the unoptimized assembly code has been eliminated by the compiler .

The later 2 statements perform the evaluation of the expressions

( i + j + 30 )   and    ( i + j + 10 ) respectively .

We have just used a very simple case of CSE , but this optimization technique could be used in even more useful cases . Consider :

k = f() + g() + 30

r = f() + g() + 10

where f() and g() are functions . In an unoptimized code , the technique of CSE wouldn't be used and hence there would be overhead due to the redundancy added with the more serious problem of having to call each function twice . The technique of Common Subexpression Elimination avoids this very redundancy  .

Function inlining

Function inlining is performed when a request has been made to the compiler to perform optimization on the code . The compiler will try (its important to remember that ,it is a request made to the compiler, not an order) and insert the complete body of the function in every place in the code where that function is used so as to eliminate the time overhead when the function is called . We could get an idea of function inlining by peeping into the assembly code generated by the compiler .

We do this with the help of a small program :

int sqr(int x)
{
return x*x;
}


main()
{
printf("%d\n", sqr(10));
}


We have the program written in a file  'pgm.c'

Do :

$: cc -S pgm.c

We could now view the assembly code generated by the compiler

Do:

$: less pgm.s

I found out the following lines in the 'main' function of ' pgm.s '

movl    $10, (%esp)
call    sqr

You would have the idea cleared up

The constant '10 '  is stored in the stack , and the call to the function 'sqr ' is made . The result of squaring up 10 is performed in the function 'sqr' . This obviously would create overheads for invoking the function .

Now consider the case where you have requested the compiler to perform all optimizations on your code .

$ :cc -S -O3 pgm.c

Now have a look into your assembly code .

$ :less pgm.s


Now just have a look into the your 'main' function and find out what you see where you had seen the previous two lines that i did mention above .

You would just find a singe line corresponding to those two lines which you had found in your assembly code that was not optimized .

This is what i found ,

movl    $100, 4(%esp)


The final value that you obtain after finding the square of 10 ( i.e 100 ) is has been moved into the stack , but where is the call to the function 'sqr' ?

You have just seen ' function inlining '. The function body had been inserted at the point where the function was used and the value '100 ' calculated at compile time instead of performing the calculation at run time .This saves a lot of time that would have been required to calculate the value of '100' by making a call to the function at run time .Performing function inlining avoids this .

Monday, 18 October 2010

Python DB-API

Python is one of the more popular Open Source programming languages, owing largely to its own native expressiveness as well as to the variety of support modules that are available to extend its capabilities. One of these modules is DB-API, which, as the name implies, provides a database application programming interface. We will have a discussion about how to connect and use a General purpose database system like Mysql with Python. The DB API provides a minimal standard for working with databases, using Python structures and syntax wherever possible .

4 Steps of the Python DB-API includes the following:
    1) Importing the API module. 

    2) Acquiring a connection with the database.

    3) Issuing SQL statements and stored procedures.

    4) Closing the connection

There are lots of database systems that are available .Here we use Mysql as our database system .You could also use database systems like PostgreSQL, SQL Server etc. Python's DB-API uses a two-level architecture in which the top level provides an abstract interface that is similar for all supported database engines, and a lower level consisting of drivers for specific engines that handle engine-dependent details. This means, of course, that to use DB-API for writing Python scripts, you must have a driver for your particular database system. Since we have fixed our database system as MySQL, DB-API provides database access by means of the MySQLdb driver . Therefore we need a module MySQLdb in Python '' .

At first , we need to get Mysql installed in our system.

$: apt-get install mysql-server-5.0

Just try and enter into the interactive prompt of mysql .

$: mysql -u 'user' -p

You will get your mysql interactive prompt where you could just brush up your sql knowledge.

Step 1) Importing the API module.


You need to make sure that you that have MySQLdb installed on your machine.

Do this:

$: python

>>>import MySQLdb

Obviously if you dont have it installed , you would be getting this :

Traceback (most recent call last):

File "<stdin>", line 1, in <module>

ImportError: No module named MySQLdb

To install MySQLdb module, download it from MySQLdb Download page and proceed as follows:
$ gunzip MySQL-python-1.2.2.tar.gz
$ tar -xvf MySQL-python-1.2.2.tar
$ cd MySQL-python-1.2.2
$ python setup.py build
$ python setup.py install

Now do

$ : python

>>>import MySQLdb

>>>

We have done with Step 1) out of the 4 steps of the Python DB-API .

Now we move into step 2). The prerequisite for performing step 2) of the DB-API is that you need to create a database ( 'Persons' in our examples to follow ) in mysql .

You could do this by :

$: mysql -u 'user' -p

Enter mysql .

Do :

mysql>CREATE DATABASE Persons;

mysql>Query OK, 1 row affected (0.03 sec)

Once you have done with this

Step 2 ) Acquiring a connection with the database.


 


conn = MySQLdb.connect (host = "localhost",

user = "user",

passwd = "asdfgh",

db = "Persons")

Here we open a database connection using the 'connect' function of the MySQLdb module.

The four parameters that are passed to the function 'connect' are :

host --      the system in which the database is created


user --      the user who is trying to acquire a connection with the database ( the user must be authenticated with mysql if the attempt to create a connection needs to be successful.

passwd --      the password of the mysql database system .

db --      the name of the database created using mysql .

If the 'connect()' call succeeds, it returns a connection object that serves as the basis for further interaction with MySQL( here 'conn'). If the call fails, it raises an exception .Therefore you better perform exception handling so that you get to know more about the root cause of the error.

try:

conn = MySQLdb.connect (host = "localhost",

user = "root",

passwd = "asdfgh",

db = "Persons")

except MySQLdb.Error,e:

print "Error occured ",e.args[0],e.args[1]

sys.exit (1)




Here on an unsuccessful attempt to make a connection , an exception is raised ( MySQLdb.Error ) , the information about the error is stored in e.args[1] .

 

Step 3) Issuing SQL statements and stored procedures.


Now we need to create a cursor object for the connection between Python and mysql so that all the functions performed by sql are performed within from Python now . We create a cursor object using the ' cursor()' method .

We are all set to execute sql statements in an Object oriented environment , and thats using the 'execute()' function . Any sql statement can be called in from Python using the 'execute()' function.

Consider you want to create a new table in our database 'Persons' .

cursor.execute("""

CREATE TABLE PERSON

(

ID int,

lastname CHAR(40),

firstname CHAR(40),

City CHAR(40)

)

""")

You have created a table named 'PERSON ' with fields as mentioned above .

As you would imagine , any other statements could be performed as such , for eg.

cursor.execute ("""

INSERT INTO PERSON (ID, lastname,firstname,City)

VALUES

(1,'KUMAR','SUNIL','KANNUR'),

(2,'JOHN','HARI','KOTTAYAM')

""")

or

cursor.execute ("SELECT * FROM PERSON WHERE City like 'K%'")

and any of the sql statements .

Step 4) Closing the connection


conn.close()

Here we have terminated the connection after all our need with the DB-API has been done with . The DB-API allows us to update , insert , retrieve and modify within a database in a simple and elegant way .

We have tried and demonstrated a simple example of how to use the Python DB-API .

Using Python with Sql,we have found an object-relational mapping where the object-oriented programming paradigm of Python meets the relational paradigm of Sql. An object-relational mapping is a bridge between the two worlds. As we saw , it lets us define classes that correspond to the tables of a database. Later , we use methods on those classes and their instances to interact with the database .

Friday, 15 October 2010

Implementing the Avl Tree

AVL Tree is a self-balancing binary search tree and it was the first such structure to be invented. The tree is said to be balanced because the difference in the heights of the child subtrees differ by atmost one. The operations Insertion, Deletion and Lookup on an Avl tree is of the order of  Log n because of this balancing act. Here we take a look into the implementation of an AVL tree .

First we need to give a thought on the term 'balance Factor' and on the major operations that are involved in the making of such a tree.

Balance Factor:Balance factor is associated with each node of the Avl tree. Its the difference between the heights of the left subtree and right subtree of that particular node.

Balance factor = height of left subtree – height of right subtree.

A balance Factor of 0 , +1 or -1 for each and every node of the tree indicates that its balanced. Whenever the balance Factor of any of the nodes equals – or + 2 , the tree becomes unbalanced and needs to undergo balancing.

Now lets just peep into the major operations that are involved in the build up of the tree.

There are 3 of them
    1) Insertion:2) Deletion:3) Rotation:

Insertion is the usual process. There is nothing special in inserting a node into an Avl tree than in inserting it to a normal binary search tree.

Deletion is again the same with a minor difference which we will look in to as we continue.

Rotation is the balancing act- The operation that distinguishes an Avl tree from the normal binary search tree. Rotation needs to be performed whenever the tree goes out of balance – this could happen as a result of the first two operations ( Insertion and Deletion ). The Balance factor may go beyond 1 and hence the tree will have to be balanced so that it remains as an Avl tree. Each of the three operations , specially that of rotation will have to be given due care in building up an Avl tree.

You could sum it up this way :

Each insertion in an Avl tree is a combination of normal insertion into a binary search tree added  with finding out whether the tree is unbalanced or not followed by rotation if necessary.

Each deletion in an Avl tree is a combination of normal deletion into a binary search tree added with  finding out whether the tree is unbalanced or not followed by rotation if necessary.

Before we get into more details of each of them , we will have a look into the way each node is implemented in the Avl tree.

Each and every node in the Avl tree is implemented as a structure .This is same as what is done in an ordinary BST , since Avl tree is itself a BST.

struct node{

int data;

struct node *parent,*left,*right;

}*p,*root=0;

The field 'data' contains the actual data that is to be stored in the node .There are three pointers of the type 'struct node*'.

*left –> points to the left child of the node

*right –> points to the right child.

*parent –> points to the parent node .



Now lets get into the details of the different operations performed on the Avl tree.

Inserting into an Avl Tree can pictorially be represented as follows:



Obviously , there are other functions that are involved in the process, but the ones shown in figure are the significant ones.

avl_insert : As discussed earlier insertion into an avl tree consists of the normal insertion followed by the checks for balancing and rotation if necessary. Normal insertion is performed first . Then the function self_bal() is called :

self_bal() : The newly inserted node would obviously be balanced since it wont have any children. Therefore the balance factors of the nodes that come above the inserted node in the tree ( starting from the parent ,right upto the root ) is to be checked. The function 'self_bal' invoked 'balance()' which performs the further tasks. Again, since a newly created tree is always balanced and since parent of the root points to NULL , the function 'balance()' should not be called if the newly inserted node is the root itself.

balance() : Invokes the function bal_fac() and checks if the balance factor is equal to +or- 2 . If yes , calls the appropriate rotate function.

bal_fac() : Finds out the balance factor for each node.

rotate() : Here rotate() is the just the cohesive name for the two types of rotate functions . According to the balance factor , the appropriate 'rotate' functions are called. We will discuss them in detail later.

The pictorial representation of 'delete()' would be no different.



By now you would have understood that the central aspect in building up the whole Avl tree is the rotate() function, but before we get into implementing the 'rotate()' function , its of prime importance that we give a little more thought into implementing the 'delete()' function in the Avl tree. As mentioned above , there is just a minor change that needs to be made , due to the self balancing nature of the Avl tree.

While performing a 'delete()' function in a normal BST, we need to look into 5 different cases of how a particular node could be deleted and replaced with another one. Just peeping into them:

1)The victim node being a leaf node.

2)The victim node having a right child.

2.1) The victim node having a right child with a left subtree.


2.2) The victim node having a right child with no left subtree.


3)The victim node not having a right child.

3.1) The victim node having a left child with a right subtree.


3.2) The victim node having a left-child with no right subtree.


Certainly , from what we have discussed until now, its clear that one of these conditions never happens. You might have guessed it by now.

Case 3.1) would never happen in an Avl tree. This could be explained with the following figure.


Consider you want to delete the node '10' after adding '9' to the tree , but adding '9' to the tree would leave it unbalanced.



Hence the tree will be rotated so that it becomes balanced ( how?, we will come to know shortly) before you could delete '10' . Hence there will never be a case of a victim node have a 'left child with a right subtree' and does not have a right child of its own.

Now , its time that we move into the  in the creation of an Avl Tree, the 'rotate() ' function.

Rotation ,as mentioned ,follows up a deletion from or an insertion into an Avl tree in such a way that it destroys the balanced nature of the Tree.

There are 4 kinds of rotation which are implemented using 2 different rotation functions.

Its crucial at this stage that the following points are understood.

Balance factor of a node = +2 implies that the node ( and ultimately the tree) is unbalanced because of the left subtree of the node and there will be one right rotation for sure.

Balance factor of a node = -2 implies that the node ( and ultimately the tree) is unbalanced because of the right subtree of the node and there will be one left rotation for sure.

Whether a single rotation needs to be done or whether the tree needs to be rotated twice depends upon the balance factor of the child of the root of the subtree that is unbalanced. You needn't scratch your head on this , stuffs that follow would lighten up things.

Let 'point' be the root of the subtree which has made the tree go out of balance . The 4 kinds of rotation are:

1)left – left rotation: If balance factor(BF) of point is 2 and the BF of the left child is > 0 , then a single 'right' rotation would do it for you.

2)left – right rotation: If BF of point is 2 and the BF of the left child is < 0 , then a 'left' rotation followed by a 'right' rotation would do it for you.

3)right – left rotation: If BF of point is -2 and the BF of the left child is > 0 , then a 'right' rotation followed by a 'left' rotation makes the tree balanced.

4)right – right rotation: If BF of point is -2 and the BF of the left child is < 0 , then a single 'left' rotation would be enough.

We will discuss rotation which happens as a result of the tree being unbalanced after an insertion or a deletion has been performed. The 'rotation' function is independent of whether the previous function was a deletion or an insertion , it depends only on the current structure of the tree.

Left-Right Rotation & Left-Left Rotation


Consider the figure :


Here the point of imbalance is '23' and the reason for imbalance is the left subtree of '23'. Hence performing a 'right' rotation is essential . Now the left child of '23' has a BF of -1 ( < 0 ) . Hence a left rotation also needs to be performed. This leads to a left-right rotation .

left rotation : Left rotation is performed by invoking the rotate_left() function. The two parameters that are to passed to the function are 'root' and 'pivot' .

Root ---> point of imbalance , I.e '17'

Pivot ---> right-child of the point of imbalance I.e '19'.


 



The left rotation has been performed.

Now our problem has been reduced to performing a left-left rotation , you could see that from the figure .That is ,we need to perform a single 'right rotation':

right rotation : The right rotation is performed by invoking the rotate_right() function. The two parameters would be :

Root ---> point of imbalance : I.e 23

Pivot ---> left-child of the point of imbalance I.e '19'.


 


 



 


The right rotation also has been performed. Now you may have a look at the tree and would find out that its balanced .

Right-Left Rotation & Right-Right Rotation.



Consider the figure above :

Here the point of imbalance is '17' and the reason for imbalance is the right subtree of '17'. Hence performing a 'left' rotation is essential . Now the right child of '17' has a BF of 1( > 0 ) . Hence a 'right' rotation also needs to be performed. This leads to a right-left rotation .

right rotation : right rotation, as mentioned above is performed by invoking the rotate_right() function. The two parameters that are to passed to the function are 'root' and 'pivot' .

Root ---> point of imbalance , I.e '23'

Pivot ---> left-child of the point of imbalance I.e '19'.



The right rotation has been performed.

Now our problem has been reduced to performing a right-right rotation .That is ,we need to perform a single 'left rotation':

left rotation : We have seen that left rotation is performed by invoking the rotate_left() function. The two parameters would be :

Root ---> point of imbalance : I.e '17'

Pivot ---> right-child of the point of imbalance I.e '23'.



The left rotation also has been performed. The tree has been balanced .We have found out how a tree could be balanced through rotation.

There is another issue that we need to deal with while performing rotation. Two questions that we need to consider here are...

1)What happens to the left node of the pivot during a left rotation.?

2)What happens to the right node of the pivot during the right rotation.?

Consider the following figure .


The tree as you could see is unbalanced with the root ('23') being the point of imbalance, the left subtree being the reason for the imbalance.. Since the BF of the left child of the root = -1 ( < 0 ) , we need to perform a left – right rotation . I.e a left rotation followed by a right rotation . For performing the left rotation , the root is '17' and the pivot is '19'. As you could see from the figure ,the left child of the pivot is marked 'L' . Our question is what exactly happens to this 'L' after the left rotation has been performed. ? The answer is right there in our next figure .


L has become the right child of '17 ' . To be more precise ,after a left rotation, 'L'( the left child of the pivot) has become the 'right child' of the node which was the parent of the 'pivot' before the rotation was performed.

As you could see, the tree still remains unbalanced as we have not performed the 2nd phase of our left-right rotation . We perform the right rotation and make the tree balanced .


I think , by now you would be able to guess out the solution for the second question that we had asked .

What happens to the right node of the pivot during the right rotation.?

The answer is certain , this is exactly the complement of what happens after a left rotation that we have just seen . After performing a right rotation , the right child of the pivot becomes the left child of the node that was its parent before the rotation was performed.

Here , I have tried and dealt with almost all issues that come up in the implementation of an Avl tree. You could download the complete source code here

Sunday, 3 October 2010

GNU make

The make utility determines which pieces of the program needs to be recompiled and issues instructions to recompile them.GNU make is the most popular make available. Lets get into the working of make with a simple example but please do read THIS before moving into further details.

By reading that, you would have understood that our ultimate aim is to produce a file '_avl.so' so that the C module 'avl.c' could be extended into Python.
Lets check what exactly do we need to do here.

We begin with 2 files namely 'avl.c' and 'avl.i'. We need to create a wrapper file at first . This is done by:

$: swig -python avl.i

Next what we need to do is that we need to create 2 object files for the respective C files- i.e we need to produce 'avl.o' and 'avl_wrap.o' from 'avl.c' and 'avl_wrap.c' respectively.
We do this by:

$: gcc -c avl.c avl_wrap.c -I /usr/include/python2.5/

The last and final step is to create '_avl.so' from the 2 object files.

$: ld -shared avl.o avl_wrap.o -o _avl.so

The dependencies and actions are quite clear now.


Typing in these commands time and again would be a tedious and ineffective job for anyone. We could automate the whole process by writing a file called Makefile and use the 'make' utility.

The Makefile for performing the above steps would be as follows:



A Makefile has the following format.



Its important to note that the 'ACTIONS' line begins with a TAB as it is a part of the syntax of a Makefile. Anything other than TAB would be harmful.

Lets check out how things work out.

$: make

This would cause GNU make to start executing the commands in Makefile.GNU make starts from the very beginning of the Makefile. The first set of 'TARGET-DEPENDENCIES-ACTIONS are checked in.

The TARGET is '_avl.so' : The DEPENDENCIES are 'avl.o and avl_wrap.o' and the ACTION that needs to be performed is

ld -shared avl.o avl_wrap.o -o _avl.so

But at present 'avl.o' and 'avl_wrap.o' are not available. To Generate them the next set of the Makefile is to be checked. The same problem exists there as well . Hence the 3rd set in the Makefile is reached where the DEPENDENCY is 'avl.i' which exists at present. Hence the action

swig -python avl.i

is executed , the target 'avl_wrap.c' is generated .
Now the second set in the Makefile can be compiled followed by the compilation of the first set. The procedure that takes place here is recursive .All the three actions are executed , the TARGETS are generated from the ACTIONS and DEPENDENCIES and finally we obtain '_avl.so'.

You could see another line in the Makefile that has not been mentioned upto now.
make remove
make remove is the TARGET and the ACTION that needs to be performed is
rm avl.o avl_wrap.o avl.py _avl.so avl_wrap.c avl.pyc

$: make remove

will remove all the unnecessary files in the directory when you are planning to start from the beginning.

If you would like to have a test on the intelligence of 'make' , you would be pretty surprised.

Lets deal that case with an example as well.

DO:

$: swig -python avl.i

We have executed the first command manually.
Now do:

$: make

and you will find that make executes only the other two commands that is included in the Makefile.
This again shows the recursive nature of the make process. The Makefile is read from the top. On reaching the second set , GNU make realizes that both the dependencies 'avl.c' and 'avl_wrap.c' are available and executes the ACTION to produce the TARGET 'avl.o' and 'avl_wrap.o'.

Now once again do
$: make

You would get a message as follows:
make: `_avl.so' is up to date.

This again brings to the fort-light the power of make. The final target '_avl.so' has been found to be 'up to date' and hence there is no question of having to execute the commands in the Makefile . GNU make, Richard Stallman and Rolland Mcgrath's creation, has recognized this.

SWIG(Simplified Wrapper and Interface Generator).

Scripting languages like python presents a lot of ease to the programmer while coding , but the fact remains that there are certain tasks that would screw you up when attempted in Python. So the coding is done in a more flexible language like C and the modules are imported in Python. SWIG is a tool that is used for extending your C programs into python and make the functions callable from Python.

Here we discuss how to import a C module into Python using SWIG.
Installing SWIG in your system ,if you haven't already, is the first task.

$:apt-get install swig

would do that for you.

Now its better you create a directory for the entire purpose and setup yourself within it.
Consider you have a C file ,avl.c , as our example (Avl is a height balanced Tree ,avl.c is an implementation of such a tree.) You could download the source code for avl.c here.

Our first step is to create an interface file 'avl.i' for setting up the interface. 'avl.i ' consists of the declarations of the various functions and global variables that are used in the file 'avl.c' and are prefixed with 'extern'.

%module avl

%{

extern struct node *root;

extern struct node *p;

%}


extern void insert(struct node* move,int item);

...
...
...
...

The 'avl.i ' file would like as above:
Now we have two files in our directory ,

avl.c & avl.i

This is all that we have to code for the task of extending a C module in Python. Now its all about SWIG.


Lets move further: Do the following.

$:swig -python avl.i

now have have a look into your directory and you will find two more files there .

avl.py & avl_wrap.c
avl_wrap.c is the wrapper file that has been created for the purpose of extension . Wrapper functions act as a glue layer between languages.

$: gcc -c avl.c avl_wrap.c -I /usr/include/python2.5/

You could now see 2 more files present in your directory.
avl.o & avl_wrap.o

These two are the object files that have been created for avl.c and avl_wrap.c respectively.

$: ld -shared avl.o avl_wrap.o -o _avl.so

This is the last thing you need to do before you could use start using your C module in Python.
Check your directory and you could see a new file , '_avl.so'. This is the shared object file that has been created .
If you create a make file for these , then obviously you wont have to go doing these tasks repeatedly.
Now ,the python module corresponding to the C module avl.c has been created which means that now you could start using it.

$: python

>>>import avl
>>>avl.my_insert(10)
>>>avl.my_insert(20)
>>>avl.traverse(1)

The source code can be downloaded here.

Saturday, 25 September 2010

Evaluation of Scheme in Python

Those who have ever gone through the SICP ( Structure and interpretation of computer programs) , would have come across the implementation of a Metacircular evaluator ( an evaluator written in a language that it evaluates) for scheme in the 4rth Chapter.


Motivated by that , and drawing ideas from the wizard book , an evaluator for Scheme is created in Python, so that Scheme statements evaluated in Python.

The creation of a metacircular evaluator for scheme contains prominently of two steps:

1) Eval: To evaluate a combination (a compound expression other than a special form), evaluate the subexpressions and then apply the value of the operator subexpression to the values of the operand subexpressions.
2) Apply : To apply a compound procedure to a set of arguments, evaluate the body of the procedure in a new environment.



Implementation of the Evaluator in Python:

Before getting into the creation and implementation details of the evaluator , please read this.
Now that you have got an idea about the concepts of environment , lets get into further details of creation of the evaluator.

The first process is basically to parse the user Input and use it in a way so as to evaluate it. After parsing ,the user input is provided to the function gravity().This particular function performs the task of classifying the user input and route them to appropriate functions. This is similar to 'eval' in the scheme metacircular evaluator implemented in SICP.
According to the user input , the appropriate functions are performed.
Different functions implemented in the evaluator.

evaluate():
This function could be compared with the 'apply' procedure implemented in SICP. The function checks for different operator and its corresponding operands. The operator is applied to the operands and the result returned.

opdet():
The function is called from 'evaluate()' , it takes in a list and returns the operator that has to be applied to the operands.



If a simple expression is provided just to perform basic arithmetic and logical operations , only these two functions are invoked after the gravity () function and we obtain the result. The situation, as is obvious would be a little different when relatively complex tasks are give ( such a processing of an 'if' statement , a function , an assignment operation, a recursive function and so on.


Processing of an 'if' statement involves invoking many more functions.

if_begin_eval():
This function involves the presence of any variables involved in the 'if' statement and fetches the value of such variables from the environment.

if_eval() :
This particular function is used to evaluate the 'predicate ' of the 'if' statement and returns the result of the 'if' statement. The function invokes the 'evaluate()' function and would process the entire 'if' statement according to the value returned by 'evaluate()' .
If a simple 'if' statement'
( if ( > 4 5 ) 4 5 ) is inputted:
if_eval() would pass the predicate part ( > 4 5 ) to 'evaluate()' and process the if statement according to the returned value. Here the value returned by 'evaluate()' would be 0, since the condition ( 4 > 5 ) is not true. The value returned by the 'if_eval()' would be 5.

if_cons_alt():
The function evaluates the consequent and alternative of the 'if' condition'. A separate function for the process would be specially helpful when the if statement consists of larger expressions and values.
The function would return back a 2 element list of the final values obtained after evaluation of the consequent and alternative.



Evaluation of procedures would take us through a different path. We need functions that can extract, evaluate and execute a function. We also need a function to test whether the inputted procedure is recursive or non-recursive.

exec_call():
Checks whether the function is a simple function of the sort :
( define ( lambda ( square ( x ) ( * x x ) ) ) non-recursive
or of the sort:
( define ( lambda ( fact ( x ) ( if ( = x 1 ) 1 ( * x fac ( - x 1 ) ) ) ) ) ) recursive :
executes the procedure if non-recursive.

extract_funcbody():
extracts the body of the function. Passes the body of the function to evaluate_funcbody .

evaluate_funcbody():
Fetches the value of the variables in the function body and returns back the function body to extract_funcbody(). This function is invoked only when the procedure is non-recursive.

exec_func_body():
Fetch the value of the variables in the function body . This function is invoked only when the inputted procedure is recursive.



Use of the module ops().
You would see a module named ' ops' being imported. This module consists of certain functions that can be performed on lists. The module is imported so as to get a feel of the way programming is done in scheme , any operation on lists can be performed on lists using 5 basic functions. The attempt has been made to stick to the combination of these basic operations rather than using the full functionality of Python.

The source code of the project can be obtained here.

Concept and creation of Environment

One of the most important thing you need to grasp while doing a project on interpreters and evaluator is the idea of an environment. Within the program the environment is a collection of the names(whether variables or functions) and its corresponding values. While using such names you will need an environment where you can store the value of those names so that you could fetch it for further use. While implementing the evaluator for scheme in python , the concept of environment was implemented using associative arrays in python. 3 dictionaries were used to implement the concept of environment. Lets go through a bit of details on the creation and use of environments.

Consider writing the scheme statements:

scheme>>>( define n 7 )
scheme>>>( define s 6 )
scheme>>>( if ( > n s ) n s )

In our 'if' statement we have not specified the exact value , but instead we have used variables. The question comes then that from where do we retrieve the values of those variables.
The answer is simple: 'Here the variables are global variables and are accessible from anywhere within the program'. Hence we need create a dictionary which consists of 'n' and 's' as its keys and their values as the values of the dictionary. This dictionary is created when the statement 'define ( not followed by a parenthesis) is inputted.

1)dict={'s':'7','n':'6'}

Obviously we cannot use this dictionary for all our purpose. We also need a dictionary where we could store variables and values that are local to a function. We have created another dictionary for this very purpose .

Consider the following:
scheme>>>( define ( sq ( lambda ( x ) ( * x x ) ) ) -----> statement 1
scheme>>>( sq 4 ) -----> statement 2

When the second statement is executed ,a dictionary 'local_dict' is created where this value 4 is associated with the variable 'x'.

2)local_dict={'x':'4'}

We use the 3rd dictionary for associating a function with its body. When the 'statement 1' above is inputted , a dictionary 'func_dict' is created which contains the 'function name' as its key and the 'body' as its value.

3)func_dict={'sq','( lambda ( x ) ( * x x ) )' }

We have functions 'create_env ()' and 'create_localenv()' for creating the first and third dictionaries that we have mentioned above. The second dictionary is created within the function 'exec_call()' where the processing of each function is started. Hence its appropriate to create the dictionary needed only for the local use of that function within exec_call().