STU199 - ECA MiniPortfolio update 2

Ive been on leave for two weeks and started back yesterday.

We had a meeting for this project yesterday with the head of registry, an ECA representative, our project manager and Mike McMonagle the developer assigned to assisting me.

It has become apparent that the business processes this application will seek to automate have not been explained clearly, we had calum from ECA down yesterday to clarify a few points.

Unfortunately there is a lot more going on manually than we had first thought, as a knock on effect this means we now have to do more work to automate them. This project has gone from a migration to rewriting many of the processes involved.

I now don't think I will be able to complete this project before I leave in august.

STU199 - ECA MiniPortfolio - signoff review

Update before I go on leave next week.

I am now near completion of the applicant side of the ECA application, the staff allocation process is casuing issues in that we are still not 100% sure of how this process is managed as some of it is administered manually.

I am working on the final stages of the applicant side and will be moving on the to final process of staff allocation shortly. The allocation procedure is where I expect the build will become more difficult as this section will need some reworking to make it run correctly, we also need to know what the manual part of this process is.

Signoff meeting

This meeting was attended by a registry representative, the heads of department for application management and technology management, my project manager, riky and Iain (department head) from dev tech.

Devtech have announced their proposal for the shared file system the application will run, this has been protested again by application and technology management on the basis of it being too complicated for them to maintain.

Mark d'amara (technology) has stated he is not satisfied that the project is out of development on the basis he does not understand the file system and it is not specified in my SDS (I understood that this system was part of infrastructure).

The meeting became quite heated, the two department heads were blocking the progress of the design. Iain (devtech) had explained the system in detail however they still held the opinion that it was too complicated for them to maintain, when asked what they would prefer they suggested manually checking all the submitted files on both servers. This raised further argument that no one would want to spend the required amount of time doing this.

It would later be explained to me that the difference of opinion could be a political one as opposed to a practical one. To me sorting through hundreds of images by hand is not practical.

We have been asked to detail this system in the SDS now before it can be signed off.

STU199 - ECA miniportfolio - Build

This week I am in the build phase of the project.

At the start of the week there were no details supplied from devtech regarding the unix accounts for hosting nor the the database logins/passwords. Chased riky up via the pm and some of this has been done.

We now have database we can connect to I can now run the development version of the application as I build it.

We don't have the file storage system up and running yet although devtech are saying that it shouldn't take any extra time since they are looking for the application to handle the file synchronisation. They never said how long though.

SDS

Mike has also finished with my design document, there are two changes required:

• How will I schedule the staff intake from EUGEX - this will be an scheduled, automated task.
  
• remove references of CRON and replace with coldfusion scheduled task - instances now changed.

He has looked over the changes and signed it off on Bills behalf. This will be used for the project signoff next week.

Build Progress

The build is actually going well, I'm not having too much difficulty with the mach-ii framework and I am able to convert the outdated functional implementations to more up to date versions.

As I've said in previous entries there are gaps in the software where manual work is being carried out and some of this still needs clarified.

STU199 - ECA miniportfolio - Design

The familiarisation period is now finished and this week I am now working on the design document. I've also had two days leave, monday and tuesday so three days on the SDS out of five.

What I have done is try my best to automate the manual processes already in the software, as the current system works reasonably well. There does appear to be holes in the processes though, some of it must be done manually around the hard coded areas. It surprises me that they are able to keep track of all the changes that need to be made in this application.

To produce the specification I have followed the business requirements document (BRD), investigated coldfusion functions and data structures and seen what would be suitable for this applications needs.

The original database schema requires some updates I am including a lookup table for the staff and course relationship and adding in a staff table. Extra fields are still being defined at the moment and I am awaiting further information on this.

I have also been consulting with mike mcmonagle, another developer here, he has been reviewing my work.

So far I have been testing methods of automation of some processes and adapting my database to suit.

I will be submitting my design for approval today and I will update when I get it back, it is likely since this is my first big design task that I may need to make a few changes.

Meeting

We also had a progress meeting today, this was more focused around the storage solution being worked on by Riky in devtech, with the servers we are using being load balance over two machines that means we needed to work out how we can store files which would be available to both machines and allow both reading and writing by each server.

This has introduced the issue of disaster recovery, how we recover this file system seems to be more difficult than than the issue of storage itself. I have been asked to try and work out a coldfusion system of recovery, this system should check both servers local storage for inconsistencies and then copy the files between machines to ensure the shared storage now holds the correct data and that none has been lost while the system was down.

ITS199 - ECA Mini Portfolio - Familiarisation

Update

I have now finished the budgeted period of familiarisation for this application.

So far I feel I have a reasonable understanding of how this application works and I feel confident that I can soon move onto the design phase of the project

In order to familiarise myself with the system I took the following steps;

Firstly I broke the system into its applications (as defined in the original business requirements and by the URL's used to navigate to them)

• Applicants
• Administrators/Staff

For each of these I drew a UML (there is every possibility it may not be perfect UML) model of the page sequence (and any deviations from it) a user would follow in making a portfolio submission. As I went through this I noted the functions each page used.

Following on from here I then went through the code starting from the Application.cfc file and began to model all the processes used in the Applicants side, where they start and what functions are involved in automating them. This model also took into consideration the datasources used and queries ran for each process.

These step were also repeated for the Admin side, although I ran into problems here as I am unable to access certain parts of the administrator functions and it seems that without getting another administrators passwords, I don't think this will be allowed as the MiniPortfolio is still a live application and so is its database. I don't have access to the ECA's servers so my access is restricted, what I do have access to is the code itself and a read only copy of the database.

The software itself has been written in a procedural style and has vaguely been compartmentalised (i.e. the admin side in an admin directory, business logic in another directory and some form of framework to the architecture). As I missed the chance with iSkills I am going to design this application to use the Mach-II framework (frameworks are the last hard skill I need to have before I leave my placement then that gives me a working experience of them).

I know I will need to add a further table to the database schema however I asked other developers to look at the original and was told its only semi relational and will need changes, at this stage its probably worth redesigning that as well. The database itself is on a Microsoft SQL Server DBMS and will be replaced with the latest Oracle 11g DBMS which is the standard for the university and will not incur further expense in terms of licences. Luckily I have some experience with both.

The issue which may arise here is the application depends heavily on the database and there are LOTS of queries taking place on each page and through many of its functions. If I am to redesign the database then this will make the existing queries obsolete as some of the tables and fields may no longer exist meaning longer development time. To compound this I only have limited experience in relational databases, my DBMS module was a long time ago now and I'll be learning on the job, first I have to find out if its worth the work as there is every chance I might not get to do it as a restructure of the database and rewrite of the existing code may exceed the allocated budget, however I don't know until I ask.

To summarise:

Completed

• Applicant side modelled
• Admin side modelled (as far as possible)
• User (movement through the pages) sequences mapped
• Database Schema mapped
• New Schema under development

ITS066 - iSkills progress

Coldfusion 9 Migration

Application - iSkills

Introduction

The coldfusion migrations have come about in order to bring our older (and unsupported) applications up to date with the current version of Coldfusion server, version 9.

The applications themselves have been written in the old Coldfusion 6 language and most are running on the content management system of the time which is Fusebox. As the applications are upgraded they must be transfered onto the mach-ii framework as this is something that IS apps are looking to do across all of their applications to maintain a standard format to the software we produce and use.

As part of the migration the applications must be as follows;

• fully compatible with coldfusion 9
• transfered onto the Mach-ii CMS where appropriate

the project brief is also in the /its066 cf9 migration/iSkills/documents folder.

At this stage there are three developers working on the project and there are also three different applications being upgraded in this batch. My application is iSkills.

iSkills was written around 7/8 years ago and is no longer supported on the cf6 platform, its fusebox framework will also need to be replaced with Mach-ii. The application was written as a resource where anybody could come and search for skills training or teaching of any offered sort.

The site allows users to search its database for resources using a number of parameters and then displays results based on their requirements. The site is not that different from the recent Edge project in terms of what the application is for.

New technology introduced

• Apache http server – Server software, although it’s used in most places and I have seen it before this is the first time I have used it for myself. I am now running apache in my dev setup to emulate real world server environments.
• Fusebox – Content management system (CMS), provides a framework to build applications around.
• Mach-ii – Another CMS does essentially the same thing in a different manner.
• Mod_rewrite – An apache module which needs to be enabled in the apache configuration file. This allows URL’s to be disguised or redirected, the point being that its making it harder for people to guess their way around your applications directories and be a bit naughty. Secondly it makes it easier for search engines to index the site, the spiders and crawlers used they use to do the indexing generally stay away from query string laden URL’s. Mod-rewrite is a separate component from your web application.

Stage 1 – Familiarisation

The old code was to be studied so that we knew how the old system was working. The steps I took are as follows:

· Obtained old code from the current live system (see issues section).

· Modify code to run on my cf9 development setup.

· Print off all pages and their related debugging output to ascertain what functions and queries were involved in each of the applications transactions. (hard evidence has been kept).

· Follow the above notes to see where and why these functions are used.

There was an initial budget of 1 day for system familiarisation, I don’t think one day was enough and initially thought that would be down to my skill level however I was reassured to see that I was not the only developer needing more time here.

The code was to be obtained from the live system, initially it was to come from MS visual source safe however it can be a hit or miss whether the full working code will be in there so the safer option was to obtain straight from live, support calls were placed through Unidesk as developers don’t generally have rights to the live servers.

Eventually (see issues section) the live code was obtained and required modifications to run locally. Mod_rewrite was a particular issue at this point because I had never experienced it and knew nothing about it, same can be said about fusebox and mach-ii.

Of course not knowing how that worked meant I never understood at first when the application didn’t work correctly and I set about making the changes necessary to run it. When I got a page of the site that needed to pass a query string in the URL I found the sight stopped responding in to commands and started falling back to its defaults.

What I was missing at the time was that the rewrite engine also rewrites the query string and acts like and interpreter between the browser and the software so when I tried passing anything more than a simple page request the adjustments I had made to rewrite were not enough to deal with the requests. A half hour meeting with the original developer sorted that one out and explained how it was working in relation to the iSkills app.

The time given was enough to get an overview of how the application was working however not enough to get deep into it. Bearing in mind I also had to learn about two CMS’s, mod_rewrite and the actual application itself, in a day.

Issues

• Incomplete code in source safe – solved by requesting the code from live (see images/uniDeskCodeRequest.jpg)
• Support argued about obtaining the code from live and took nearly a week to pick up the request.

Stage 2 – Impact analysis & Initial Meeting

A meeting was called to discuss the following:

• the applications suitability for mach-ii
• the amount of time we require to make the upgrade
• any issues we should consider across the whole migration project.

Please see bill lee's email dated 03/11, this is the initial invitation and explains what was required of me at the meeting.

We were required to produce a short document outlining our analysis and intentions for the build (this document is in /its066 cf9 migration/iSkills/documents). In this document I outlined the following;

• The applications mach-ii suitability.
• My view on my ability to bring the project in within the budget.
• My concerns over the more advanced levels of code (versus my skill) in the application and what it its impact would be on the time I would take to complete the project.
• The benefits of this project in terms of IS's aim to implement a common structure to the universities applications.
• the importance of this project to my development (in terms of CMS frameworks)

My proposal was accepted and the tasks were assigned (see bill lee's email dated 16/03) with the original time scales noted which it was agreed could be adjusted if the project needed it to be. its worth noting at this stage that this was the first batch of migrations to go ahead and all other migrations would follow the template that was set by this initial run.

Stage 3 – Migrate code (Build)

Not a lot of work was required to convert the code to CF9, mainly outdated coldfusion implicit functions needing replaced with more modern equivalents.

However the mod_rewrite issue persisted here, the build went fine through making of the home and search pages, I had not yet started on their functional components yet, I was able to have the pages active and looking like the did on live however when it came to passing parameters around it became a different story. When I tried to click into the third page which was the browse page you are presented with a list of categories on the left of the page, click them and that categories search results are displayed on the right of the page (see images/iskillsBrowseResults).

In order for the application to search for and display those results it has to pass a query string through the url which mod_rewrite has to interpret for the browser to interact with the software. Unfortunately this meant rewriting the first set of pages I done.

At this stage to get the results list working I came across the attributes scope of fusebox, in this application it had been used as a catch all scope which is available throughout the whole application which posed a problem. How could I find what was in that scope if I couldn’t debug the original program? Also if I was able to do so how would I replicate that in mach-ii since that scope doesn’t exist?

It became clear that this would not be a case of copying over chunks of code if the environment they were designed to be in was no longer going to be used.

In order to help me overcome this issue I requested a meeting with the original developer. He explained that the rewriting url was heavily woven into the application as well as the attributes scope, between us we felt that the time needed to redevelop the software in the mould that the project wanted would have been a lot more than was budgeted for.

I took this to the project and resource manager and the mach-ii part of this build was dropped. I was very disappointed at not being able to complete the mach-ii build but on the upside I was confident in making that decision due to what I had learned about software development with Coldfusion.

What bill explained to me was that it was less a reflection of my ability but an indication of my ability to find a potential problem and not just keep butting up against it. knowing where the limits are when it comes budget and timescale.

Iskills is now ready for peer testing, at the time of writing this it is waiting on the development server to be tested, I still have configuration work to do regards to setting up server mappings and datasources, these are actually an integral task to the team setting up the server environment however its not been done and one of the servers is currently serving up error pages which I need someone else to fix. I’m just waiting for the go ahead.

Testing

As the site is divided into two sections, user & admin, I will be testing each section as they are completed, hopefully this should lead to me having a finished and tested application before moving on.

Each section would then be devided into their respective pages and functions which again will be tested on completion.

Although I don't hold a full knowledge of it I believe this leans slightly towards a test driven development methodology, or at lest my interpretation of it.

Issues

• Tight coupling between application, framework and server. Separation meant massive rework.
• No supporting documentation available to assist in analysis and debug of old system.
• Unable to complete a mach-ii build and consider this framework to be the most important stage of this placement so far.

General Update

Hi

There are a number of things going at the moment and due to outside circumstances which have been beyond my control I have been on leave at intermittent intervals over the last month and just getting myself back up to pace now.

Projects active at the moment are:

• Edge - ITS032 (still)

• CF9 Migration - ITS066
  

I will write a catchup to introduce the current migration for ITS066.

Edge

The edge application has been deployed, the url has not yet been disclosed as the service has not formally been launched at this time.

Prior to me going off the application had been brought up to a stage where it was ready to go on the server and be user tested. During the test some issues were identified (please see darcey gillies email dated 24/03 and points 1 to 4 of the attached issues document) which have since been addressed, additionally more requests have been made for off spec changes to be included (point 4 in the above list) and at this stage the project manager is looking for somewhere to draw a line in the sand.

Prior to making the changes our progress was again hampered by confusion over its location (see alex carters email dated 16/03) , I was advised the application was stored on the T drive as per the mapping on alex's email, unfortunately it wasn't there and was put into O drive instead.

A knock on effect of this was that I had no permissions to access this drive which prevented me from doing any configuration on the application, such as the untested automated login system needing to be in the right environment to work. Permission for access was not actually granted to me for another week which was obviously a massive delay to the client.

As of yesterday the stage we are at is that this application should have been wrapped up weeks ago, we have encountered budget problems, clients changing their minds and requesting new features after the build stage and numerous delays in getting access to locations or resources.

We are now looking to add functionality to check for broken links (again the issues document above point 4) however I have been assured this is not much to do.

We will then be looking to retest and redeploy, hopefully this will close of this project.