This lengthy document will take you through the basic code of the sequence/psipred job type in the webserver. This will take you through adding a new analysis type to the webserver. At a minimum you will need to edit the View, the controller, maybe the job model and the job_status_handler. You will need some familiarity with HTML, CSS, JQuery, Ruby, Rails, XML-RPC and JAVA.

You will absolutely have to have access to the CS Dept. typo3 instance for the bioinformatics pages. In typo3 you will find the static files for the webserver including the Javascript, CSS and images the pages use. You'll find these under FileList->templates->

Adding a new analysis to the psipred front page#

1) Open up View->Psipred->index.html
This is the front page of the http://bioinf.cs.ucl.ac.uk/psipred/ it should be fairly self explanatory, the only thing to really note is the syntax for the JQuery-UI tabs widgets. These control how many tabs there are and which are being shown.

<ul>
      <li id="tabs-1-tab"><a href="#tabs-1">Input</a></li>
      <li id="tabs-2-tab"><a href="#tabs-2">Sequence Filter</a></li>
      <li id="tabs-3-tab"><a href="#tabs-3">MSA Options</a></li>
      <li id="tabs-4-tab"><a href="#tabs-4">DISOPRED</a></li>
      <li id="tabs-5-tab"><a href="#tabs-5">BioSerf</a></li>
      <li id="tabs-6-tab"><a href="#tabs-6">DomPred</a></li>
</ul>

2) Further down the page is the slector the User can use to select analysis methods. Note that the toggletab() method controls which tabs can be seen, this function can be found in the page's javascript (via typo3)

<tr><td><%= check_box_tag(:program_psipred, "1", true, { :onclick => "toggleTab(0);", :class => "program_psipred2" } ) %><%= hidden_field_tag :program_psipred, "0" %>PSIPRED v3.3 (Predict Secondary Structure)</td>
<td><%= check_box_tag(:program_disopred, "1", false, { :onclick => "toggleTab(4);", :class => "program_disopred2" }) %><%= hidden_field_tag :program_disopred, "0" %>DISOPRED2 (Disorder Prediction)</td></tr>
<tr><td><%= check_box_tag(:program_mgenthreader, "1", false, { :onclick => "toggleTab(0);", :class => "program_mgenthreader2" } ) %><%= hidden_field_tag :program_mgenthreader, "0" %>pGenTHREADER (Profile Based Fold Recognition)</td>
<td><%= check_box_tag(:program_svmmemsat, "1", false, { :onclick => "toggleTab(0);", :class => "program_svmmemsat2" } ) %><%= hidden_field_tag :program_svmmemsat, "0" %>MEMSAT3 & MEMSAT-SVM (Membrane Helix Prediction) <div class="update" >Updated!</div></td></tr>
3) Below that region there are a further range of div elements for each tab, should be fairly self explanatory.
4) Adding a new method requires adding a new method to the list in point 2). TWith a new name of the form :progam_ANALYSIS name, if it needs a new tab of options that must be added to the list from point 1) at the top of the page. Then a new div for that tab needs to be created and filed. The tabs are numbered to prevent collisions with the front and backend tabs as the javascript function runs both pages DO NOT choose a number that collides with one already in use. This functionality is described in the JQuery-UI docs JQuery-UI

Editing the controller#

1) Now we need to edit the psipred controller, Open the file in Controller->psipred_controller.rb 2) There are a number of functions here index, submit and results are really the important ones. Each function starts with a range of items that can be set externally by the SMART LINKS API. You don't really have to do anything to those unless you're adding something you want an external resource to be able to preset. Here's that block of code; the eagle eyed among you will notice that these correspond to some of the named items in the index form above

#Allow external links to auto fill some features.
#TODO: Roll this out to the other services
@sequence = params[:sequence]
if @sequence.nil?
  @sequence = ''
end

@type = params[:program]
if @type.nil?
  @type = 'psipred'
end

@address = params[:email]
if @address.nil?
  @address = ''
end

@subject = params[:subject]
if @subject.nil?
  @subject = ''
end

if params[:complex].nil?
  params[:complex] = "true"
end
if params[:membrane].nil?
  params[:membrane] = "false"
end
if params[:coil].nil?
  params[:coil] = "false"
end

3) Index isn't a very interesting function so moving down to Submit, this is where the action is running through it we start with the above block of code. The next block sets some variables for our job table from series of the incoming form elements

#create a new job object
@job = Job.new()

#Assign the basic job information from the input form
@job.QueryString = params[:sequence]
@job.QueryString.gsub!(/\r/,"")
@job.address = params[:email]
@job.name = params[:subject]
@job.Type = "seqJob"
@job.UUID  = UUIDTools::UUID.timestamp_create().to_s

next there is a lengthy section where we set the variables which define which analyses to run, that block of code begins with. This and the following logic is where you have to add your new analysis type of the form program_ANALYSIS from above

@job.program_psipred = 1
@job.program_disopred = 0
@job.program_mgenthreader = 0
@job.program_svmmemsat = 0
@job.program_bioserf = 0
@job.program_dompred = 0
@job.program_ffpred = 0
@job.program_genthreader = 0
@job.program_mempack = 0
@job.program_domthreader = 0

Afterall that we grab the user's IP address

#@job.ip = request.remote_ip, find out from where our user has submitted their job
tmp_ip = request.env["HTTP_X_FORWARDED_FOR"]
if(tmp_ip =~ /(\d+\.\d+\.\d+\.\d+)$/)
  tmp_ip = $1;
else
  tmp_ip = request.remote_ip
end
@job.ip = tmp_ip

If they want a modeller job we test the modeller key they passed in (testModellerKey(@mod_key)), then we set their User ID for job priority (setUID(@job.address)). Then we count the number of jobs they have running (countJobs(@job.ip))

Then we want the job configuration for the job requested (seqJob) from the configurations table

curconfig = Configuration.find_by_name(@job.Type)
@job.configuration_id = curconfig.id

This is followed by a block of code that tests if the incoming sequence is an MSA or not and removes some illegal/unnecessary characters

@msa_found = 0
if(@job.QueryString.to_s.count('>') <= 1)
  #remove any FASTA description line that starts with a >
  @job.QueryString.gsub!(/^>.+\n/,"")
  #remove newlines or space characters from the incoming query sequence.
  # and upcase it
  #puts @job.QueryString
  #puts @job.QueryString.gsub(/\n|\s/, "")
  @job.QueryString = @job.QueryString.gsub(/\n|\s/, "").upcase
else
  @msa_found = 1
  input_msa = @job.QueryString.to_s
  lines = input_msa.split(/\n/)
  new_msa = String.new()
  #Here we rewrite the fasta headers so that the are all unique, blast+ will cat subsequent sequences with identical headers together
  #if the header lines are not unique. The alternative would be to bounce inputs that have identical sequences/headers but maybe someone
  #has a reason for listing the same sequence twice in their MSA.
  lines.each_with_index do |line,index|
    if(line =~ /^>/)
      line.gsub!( /^>.*/, ">sequence_"+index.to_s )
    end
    new_msa = new_msa + line + "\n"
  end
  @job.QueryString = new_msa
end

This is followed by the standard Rails respond block that chooses which view to send to the user, a number of test return an error to the user

if @model_test == 0
  @error_message = "Modeller Licence Key invalid"
  format.html { render :action => "error"}
elsif(job_count >= 5 && @job.user_id != 0)
  @error_message = "Your IP address has 5 live jobs running.  Please wait until your jobs have finished before submitting more. "
  format.html { render :action => "error"}
elsif(@job.user_id == 0 && job_count >= 20)
  @error_message = "Your IP address has 20 live jobs running.  Please wait until your jobs have finished before submitting more. "
  format.html { render :action => "error"}
elsif(@job.user_id == 11)
  @error_message = "Your IP address has been banned from submitting jobs due to misuse of our server. If you feel this is an error or you wish to restablish access please contact us."
  format.html { render :action => "error"}
else

If we are not showing and error there are a number of functions that submit the job and tell the user of success:

@job.save! This saves all the verified data to the jobs table

assignOverrides() This assigns any options from the input form that may override the defaults in the configuration

assignCacheOverrides() This gets any cached PSIBLAST checkpoint file from the database

@job.submit_jobs(@job) This actually submits the job to the backend

format.html And this shows the user the submit.html page from the views

Add a Migation#

1) You've just added a new analysis type called program_ANALYSIS to the indec view and submit function of the psipred controller. Of course that type doesn't exists in the jobs database yet. So now you have to add a migration to the migrations that adds that column to jobs!

Tinker with the jobs model#

1) Add any validation that your new analysis needs to validate in the form. What's the max seq length for this method, are MSA's allowed etc....
2) Open up job_status_handler and add all the new codes for the datatypes your new analysis will be returning. Possibly you'll know beforehand and you can do this now. Usually I'd wait until I'd added my new job type to the backend as you usually don't know what all the files will be until you get to running the job.

Change NewPredServer#

1) Open up org.ucl.newpredserver.seqJob.java 2) In the run() method start by adding your new program_ANALYSIS type to the set of config options at the top

//here we should adjust the settings
int psipred_control = Integer.parseInt(m_config.getSetting("program_psipred"));
int svmmemsat_control = Integer.parseInt(m_config.getSetting("program_svmmemsat"));
int genthreader_control = Integer.parseInt(m_config.getSetting("program_genthreader"));
int mgenthreader_control = Integer.parseInt(m_config.getSetting("program_mgenthreader"));
int domthreader_control = Integer.parseInt(m_config.getSetting("program_domthreader"));
int disopred_control = Integer.parseInt(m_config.getSetting("program_disopred"));
int dompred_control = Integer.parseInt(m_config.getSetting("program_dompred"));
int bioserf_control = Integer.parseInt(m_config.getSetting("program_bioserf"));
int ffpred_control = Integer.parseInt(m_config.getSetting("program_ffpred"));
int mempack_control = Integer.parseInt(m_config.getSetting("program_mempack"));

The next bit then replicates the logic at the top of the Submit function in the PSIPRED controller, this ensures that nothing can accidentally send an odd job that somehow runs nothing.

3) Assuming your new analysis doesn't just recombine the old analysis types you need to initialise a new instance of your task instance by adding a line to the following block

runPfilt pfiltJob = new runPfilt(m_config,m_cleanupGroup, m_seq, fFasta, this);
runBlast blastJobs = new runBlast(m_config,m_cleanupGroup, m_strJobExt, m_psipool, this);
runPsipred psipredJob = new runPsipred(m_config,m_cleanupGroup, m_strJobExt,this);
runMemsatsvm svmmemsatJob = new runMemsatsvm(m_config,m_cleanupGroup, m_strJobExt, this);
runDisopred disopredJob = new runDisopred(m_config,m_cleanupGroup, m_strJobExt, this);
runDompred dompredJob = new runDompred(m_config,m_cleanupGroup, m_strJobExt, m_seq, m_psipool, this);
runMempack mempackJob = new runMempack(m_config,m_cleanupGroup, m_strJobExt, this);
runGenthreader genthreaderJob = new runGenthreader(m_config,m_cleanupGroup, m_strJobExt, this);
runMGenthreader mgenthreaderJob = new runMGenthreader(m_config,m_cleanupGroup, m_strJobExt, modelCtrl, this);
runDomthreader domthreaderJob = new runDomthreader(m_config,m_cleanupGroup, m_strJobExt, this);
runBioserf bioserfJob = new runBioserf(m_config,m_cleanupGroup, m_strJobExt, m_seq, m_psipool, m_nJobID, this);
runFfpred ffpredJob = new runFfpred(m_config,m_cleanupGroup, m_strJobExt, m_seq, this);

4) Below that are a series of try-catch blocks that try to run each analysis type, add a try catch block for your analysis type and generate a new kind of error return

UpdateStatus(StatusClass.ANALYSISError, StatusCode.JobRunning, ex.getMessage(), strError);

AND

UpdateStatus(StatusClass.ANALYSISError, StatusCode.JobRunning, (String) hBlastMessages.get("ANALYSISMessage"), strError);

You'll have to add a new StatusClass for ANALYSISError to org.ucl.shared.JobStatus.

5) in org.ucl.servertasks you need to add a new class to run your analysis runANALYSIS.java is probably a good name for it, then it can match what you added in point 3)

6) runPSIPRED.java should be fairly self explanatory. The class constructor sets all the file names and configurations settings that the job needs. run() runs each executable in turn. Then the methods runEXE() run each of the executables with a fairly repetitive process call layout.

7) The anatomy of a typical process call:

protected void RunPsipredViewer(String strPsipredViewer, String rootUUID, String m_strJobExt, String strExtension, File fSS2, File fPostscript) throws Exception
{
       try
    {
        String strCurCmd = strPsipredViewer +" "+ fSS2.getCanonicalPath();
        ExternalProcess epPV = new ExternalProcess(strCurCmd.split(" "));
        System.out.println("    Running PsipredViewer\n    " + strCurCmd);
        int nRes = epPV.call();
        Thread.sleep(10000);
        String strRes = epPV.getOutput();
        if (nRes != 0 || strRes == null) {
            //job.UpdateStatus(StatusClass.Error, StatusCode.JobRunning, "No postscript file was produced", strError);
            localJob.UpdateStatus(StatusClass.FlowControl, StatusCode.JobRunning, "PsipredViewer failed", strError);
            throw new Exception("Error calling Psipred Viewer");
        }
        Job.strToFile(fPostscript.getCanonicalPath(), strRes);
        //check the ps file is there and move it.
        localJob.UpdateStatus(StatusClass.PostscriptResult, StatusCode.JobRunning, rootUUID + m_strJobExt + strExtension, strRes);
    }
    catch(Exception ex)
    {
        //job.UpdateStatus(StatusClass.Error, StatusCode.JobRunning, "PsipredViewer call failed", strError);
        localJob.UpdateStatus(StatusClass.FlowControl, StatusCode.JobRunning, "Psipredviewer call failed", strError);
        Logger.getLogger(runPsipred.class.getName()).log(Level.SEVERE, strError, ex);
        throw (new Exception(ex.getMessage()));
    }
}

Here's a line by line break down of what we're doing.

i)

String strCurCmd = strPsipredViewer +" "+ fSS2.getCanonicalPath();
First off we build a string of the command we'd like to run as an external process (a piece of C code, a shell scritp whatever)
ii)
ExternalProcess epPV = new ExternalProcess(strCurCmd.split(" "));
We use the string in i) to build a new instance of the ExternalProcess object
iii)
System.out.println("    Running sspred_avpred\n    " + strCurCmd);
This echos some information about what our server is doing to the STDOUT where the server is running or to the server logs. It also reports what command it thinks it's running for debug purposes. THIS IS ESSENTIAL
iV)
nRes = epPV.call();
We use the call method to submit and run the external process. The exit status of the process is held in nRes.
V)
thread.sleep(10000);
Because file IO is buffer based we wait here to ensure that any STDOUT buffer we need to read will definitely be filled before we read the buffer and move on.
vi)
String strRes = epPV.getOutput();
We create a new string (strRes) for our results that the external process passed to STDOUT. Calling the getOutput method on our externalProcess object fills the string with whatever went to STDOUT
vii)
if (nRes != 0 || strRes == null)
If the exit status was erroneous OR we picked up no input then we move into the braces to throw an error
viii)
localJob.UpdateStatus(StatusClass.FlowControl, StatusCode.JobRunning, "PsipredViewer failed", strError);
throw new Exception("Error calling Psipred Viewer");
We use FlowControl to add an error to the database and then the exception thrown to be handled by seqJob.java which will pass the error back to the runner at the frontend
ix)
Job.strToFile(fPostscript.getCanonicalPath(), strRes);
If there were no problems then we save our processes STDOUT to a file, for the next executable to make use of
x)
localJob.UpdateStatus(StatusClass.PostscriptResult, StatusCode.JobRunning, rootUUID + m_strJobExt + strExtension, strRes);
And then update the front end database with the results we collected. Where possible in your new job/process type try to use a statusCode that already exists, there are status codes for a wide range of datatypes in StatusClass.java

The Results#

With any luck you've now changed the index page, updated the psipred controller, added any migration you need to add columns to job, updated the runner/daemon code to handle all the new job status classes (job_status_handler). You've edited seqJob.java correctly and added your own analysis type (runANALYSIS.java). Now you can update the results pages

1) Now you can open Views->psipred->results.html.erb
2) Again we're looking at a JQuery-UI tab table. This time we only show tabs for analyses we ran, so the logic tests each of the program_ANALYSIS types that might have been run and adds the relevant tabs. Note how they are sequentially numbered NOT to collide with the numbering in the index page.
3) After the tabs declaration, there is a chunk of code that loads jalview lite
4) Then there are the divs for the tabs. Tab 11 is the Summary page, it displays any errors that were passed from the back end by rendering layouts/backend_errors
5) It then renders the sequence digram by calling layouts/aminoacid_map
6) Next the Sequence Selector widget is called with layouts/sequence_selector, you can edit this widget to add your new analysis type
7) Lastly it renders any genthreader diagrams
8) After that we build any tab that will contain results or diagrams that are specific to an analysis type
9) The final tab gives links to download all text files and diagrams


Implementing a new Web Service for your new method#

Careful there, you probably thought you were done but...

1) Having implemented the webserver you now need to make a SOAP API for it. The ruby framework currently has datanoise-actionwebservice installed to provide this functionality. First off in the apps/api directory you need to make a new api class for your service. This defines the functions offered and the inputs and outputs, available data types are the standard SOAP 1.2 types. Take the psipred examples. The file is called api_psipred.rb, you should name your file in the format api_NEWSERVICE.rb

class ApiPsipred < ActionWebService::API::Base

  api_method :submit, :expects =>   [{:sequence=>:string}, {:email=>:string},{:name=>:string}, {:complex => :string}, {:membrane => :string}, {:coil => :string}],
                      :returns => [SubmitMessage]
  api_method :result, :expects =>   [{:job_id=>:string}], :returns => [PsipredResult]\\

end

2) All our services submit() functions return the same SubmitMessage formatted message but they have different output. Next you roll your output data structure, the file should be called NEWSERVICE_result.rb. Again the allowed data types are the SOAP 1.2 standards. Here's the PSIPRED one

class PsipredResult < ActionWebService::Struct
  member :message, :string
  member :job_id, :int
  member :state, :int
  member :psipred_postscript, :string
  member :psipred_results, :string
end

message, job_id and state MUST be included after that you set a whole load of outputs, typically strings, for the data URIs you will be returning

3) Next you have to register your new api in the psipred_api.rb controller. Open that and you'll see a handful of obvious commands for declaring and instantiating and instance of each API. Add yours to the bottom of the list

web_service :api_NEWSERVICE, ApiNEWSERVICEService.new()

At the bottom you also need to pass the requesting IP address to the model of your web service

ApiNEWSERVICEService.client_ip = @tmp_ip

4) Lastly now you need to create your model for this service. For some reason when using layered apis the whole set up is ass backwards and your controller type function end up in the model. All our services just instantiate 2 functions submit() and result(). submit() sends the job to the server and on success returns a job ID to the user. result() takes a job id and returns the state of the job and any results URIs should the job have finished successfully. These commands are effectively identical to the same commands in controller for the service you created for the html view. There are a couple of changes to watch out for but copying the api_psipred_service.rb would not be a bad idea for a service that takes a sequence (or the api_metsite_service.rb for a service that takes a pdb file). Watch out for the following, we get the client's IP address by setting the following (see step 3)

@job.ip = ApiPsipredService.client_ip

The exception catch for the job.submit! is what grabs the error message this time rather than using the @errors thing

@state = 0
  if @job && ! @job.errors.empty?
    errors = @job.errors.collect { |msg| msg }
    errors.each do |error|
      error.each do |entry|
        if entry !~ /QueryString/
          @message= @message+entry+". "
        end
      end
    end
  end

In the results function we are no longer calling a view so we grab the results here and add them to the outgoing SOAP response. Start by creating an instance of the result class we defined in 2). Then set all the output strings to "No data". Then we loop through the @job_results and if we relevant results then we over write the "No Data" strings we set just before the loop

results = PsipredResult.new
 results.psipred_postscript = "No data"
 results.psipred_results = "No data"
 if(job_complete == 1)
     results.message = "Success"
     results.job_id = job_id
     results.state = 1
     @job_results.each do |job_state|
       if (job_state.status_class == 11 && @job.Type =~ /psipred/)
         results.psipred_results = "http://bioinf.cs.ucl.ac.uk/bio_serf/getresultattached/" + job_state.id.to_s
       end
       if (job_state.status_class == 12 )
         results.psipred_postscript = "http://bioinf.cs.ucl.ac.uk/bio_serf/getresultattached/" + job_state.id.to_s
       end
     end

Add new attachment

Only authorized users are allowed to upload new attachments.
« This page (revision-1) was last changed on 13-Feb-2013 15:59 by UnknownAuthor