Character set problems with Ajax requests when using serializeJSON

Just a quick post while I'm thinking about it and for future reference about character set issues you can get when making an Ajax request and getting JSON back from ColdFusion via serializeJSON.

I am working on an item selection tool that takes a search term, submits it via the JQuery ajax method and ColdFusion does a d/b search and returns the results as JSON via the excellent serializeJSON method.

I noticed that the results coming back had numerous odd looking characters where the character being sent back wasn't rendering properly on the front-end. I have seen this many times before and it's usually the character set being used that is the issue.

It was here as well, but the fix wasn't what I expected. I thought throwing in a:

<cfprocessingdirective pageencoding="utf-8">

..on the view which returns the JSON would do the trick but it didn't in this case.

The fix was to place this at the top of the view:

<cfcontent type="application/json;charset=utf-8">

The cfcontent tag was already there as I needed to tell CF what format I wanted to return but the extra charset parameter did the trick on fixing the character rendering problems.

A useful thing to remember when dealing with Ajax calls.

Setting up Mylyn in ColdFusion Builder 2

NOTE: IF using ColdFusion Builder 2 - Update 1 you MUST use the following update site (not the helios one mentioned in the post:


For a new project I have just started we are using the excellent Mylyn task focused system for Eclipse

I have just spent a little bit of time working out exactly how to install Mylyn using the update system built into Eclipse and thus ColdFusion Builder 2.

To save other people this time, here is a quick guide to a base install:

 

Step 1: Go to the Help -> Add New Software link in ColdFusion Builder 2

 

Step 2: Type the Eclipse Helios update URL into the 'Work with' text field:

(This is the update site for Eclipse 3.6 which ColdFusion Builder 2 is built on)

... and click 'Add'. Give it a name (Helios Update Site for example) and click 'OK'

 

Step 3: Once the updates load (it may take a while) expand the 'Collaboration' item

 

Step 4: Select the following items - as shown in the screenshot:

 

Please Note: I have selected the 'Trac' connector as this is the system we are using on our current project. Obviously do not install this if you're not using Trac and select 'Bugzilla' if this is your bug tracker of choice.

Click 'Next' and then 'Finish' to install Mylyn.

Once complete you should click the button to restart ColdFusion Builder 2.

The Eclipse docs recommend starting with the -clean command line argument after installing new software which you can do like this:

 

C:\Program Files (x86)\Adobe\Adobe ColdFusion Builder 2 Beta\CFBuilder.exe" -clean

 

How I Got Started in ColdFusion

A couple of weeks ago Steve Bryant posted a blog entry with a great idea. This idea was that on 1st August 2011 all ColdFusion developers around the world should post on their blogs and tell the story of how they got into ColdFusion. As he points out on his blog post, these stories are often very interesting and not always what one might expect.

Now, I'm not sure my particular story is that interesting but I'll go a head and post it anyway. I've had a total blast working with this awesome web development system and would like to share how I discovered it.

The story begins in 1998 when I got my first job after graduating from university. It was with a small web development company who had recently lost their lead developer (to Netscape of all companies) and they needed a young graduate to take over. I got the role which was somewhat of a 'do everything' type position. My first 3 months goals were to pick up the HTML web development of the companies various clients and to learn the entire Microsoft server product suite that was in use on the internal company network.

The early days of the role were pretty much all HTML - often developing huge websites for fairly high profile clients and using Microsoft Frontpage 97. I can still recall the pain of copying pages and pages of product descriptions from technical order books, along with tricky HTML tables layouts for specifications etc. Back then, the only dynamic elements we would use were CGI e-mail handlers.

My first foray into web programming was actually not for the web. I was tasked with writing a log file analyser for Checkpoint Firewall 1. I hadn't used Perl before but it seemed like the obvious choice to use for building such a tool. It was quite a learning experience, to have deadlines to meet and a whole new language to learn but it was a great way to begin to get into more advanced development.

After that we would sometimes get asked for more dynamic elements for websites - a questionnaire for instance - and Perl was again the obvious choice, but was always quite a trial to get working exactly right.

So how does ColdFusion come into this then? Well, around 2000 / 2001 our company took on a new developer who was tasked with building a large e-commerce site. Before this the only e-commerce site we had built was via a product called Drumbeat (does anyone remember that?). It was truly awful. It built ASP websites using a visual editor and lots of code libraries but was a total pain to customise.. Not a nice experience.

Anyway, the new guy came to the company with some experience of a product called ColdFusion. He told us it was a really great way of developing websites and that he had picked up the basics very quickly and he came from a non-technical background. To speed up the development he looked around and found a product called CFWebstore which became the core of the first big e-commerce site the company had worked on.

This particular developer did not actually stay at the company very long and this project fell to me. It wasn't the best project to work on but was a good introduction to the awesomeness of ColdFusion 4.

The next developer to be taken on as a replacement had a lot more ColdFusion experience and came to the company with a number of development ideas which revolved around bespoke development - rather than off the shelf packages.

This was a real eye opener to me as he introduced the concept of custom CMS development using a MS Access DB (crazy to think we used that, though we moved onto SQL Server down the line). Sharing code meant that I was able to develop my knowledge as well as move in a personal direction in my own development. Due to the fact that we were in a small team and pretty much being given sole responsibility of each project meant that we both developed our own bespoke CMS systems, while sharing concepts with each other - always a great way to bounce ideas around.

It wasn't long before I was developing a huge range of sites ranging from basic CMS systems all the way up to enterprise Intranets and e-commerce sites. Our company never focused on one particular niche so we had to have a codebase that could adapt to *any* project. Having a bespoke CMS that could be customised based on ever changing project needs was pivitol and ColdFusion made it so much easier than it could have been. I would sometimes be shocked just how flexible CF could be when having to quickly respond to customer requirement or change requests.

I worked at that company for almost 10 years and ColdFusion became a very faithful and reliable friend... As it continues to be every day..

XMLSearch - No Results Found / Empty Array Solution

Just a quick post while it's fresh on my mind as I'm sure this is going to crop up in the future as I don't often handle XML in Coldfusion.

Basically here is the scenario. You have an XML file which isn't that complicated in structure but whatever you do XMLSearch just won't return the results you expect - it appears broken.

Here is an example from the Spotify API:

<?xml version="1.0" encoding="utf-8"?>
<artists xmlns:opensearch="http://a9.com/-/spec/opensearch/1.1/" xmlns="http://www.spotify.com/ns/music/1">     
     <opensearch:Query role="request" startPage="1" searchTerms="Ash"/>
     <opensearch:totalResults>95</opensearch:totalResults>
     <opensearch:startIndex>0</opensearch:startIndex>
     <opensearch:itemsPerPage>100</opensearch:itemsPerPage>

     <artist href="spotify:artist:2evydP72Z45DouM4uMGsIE">     
          <name>Ash</name>
          <popularity>0.72515</popularity>
     </artist>

     <artist href="spotify:artist:77zwstbi3x1IxnbDFg6uns">
          <name>Wishbone Ash</name>
          <popularity>0.67812</popularity>
     </artist>
</artists>

Ok, in the above let's say I wan't to get all artists from the XML using xmlSearch.

The usual way would be to specify the following:

<cfset xml = xmlParse(xmlString)>

<cfset artists = xmlSearch(xml,"/artists/artist")>

This *should* work fine - top level 'artist' node, followed by some 'opensearch' and 'artist' nodes. We should now have an array of artist nodes in our artists variable. But we don't. Why?

Well the simple answer is, look closely at the artist node. Notice the xmlns attributes? They define an XML namespace and throw the whole thing out of whack. Ok, it's part of the spec but to get around this with XMLSearch we need to change our call to the following:

<cfset artists = XMLSearch(xml,"/:artists/:artist")>

Simple change but this tells the command to ignore the namespace and just return all 'artist' nodes that sit underneath the 'artists' node.

A simple distinction but one that can save hours of hair tearing. :)

How to do ManyToMany Relationships with a Lookup Table Sort Order Field in Transfer

I recently had a use case in an application I'm writing where I needed to be able to sort some items which were referenced in a lookup table - modelled using ManyToMany in Transfer.

The relationship maps users to pages. It's possible to allocate the same pages to multiple users.

I initially had this relationship defined as follows (part of the 'users' object - not shown):

<manytomany name="Pages" table="lnkUsers_Pages" lazy="true">

     <link to="users.user" column="lnkIDUser"/>
     <link to="content.page" column="lnkIDPage"/>

     <collection type="array">
          <order property="id" order="asc"/>
     </collection>
</manytomany>

This all works fine and I'm able to call getPagesArray() from the users object and retrieve an array of page objects.

However, I also have the requirement to sort the returned pages based on a numeric 'sort order'. If the sort order was based upon the content pages - i.e. all pages would be sorted in the same way regardless of user allocation I could model this by modifying the collection:

<manytomany name="Pages" table="lnkUsers_Pages" lazy="true">

     <link to="users.user" column="lnkIDUser"/>
     <link to="content.page" column="lnkIDPage"/>

     <collection type="array">
          <order property="SortOrder" order="asc"/>
     </collection>
</manytomany>

 

This would sort the collection based upon the 'SortOrder' field in the content table.

However, my requirement is to be able to sort the results based upon a sort order field in the actual lookup table - 'lnkUsers_Pages'. The database table looks like this:

lnkIDUser
lnkIDPage
SortOrder

Unfortunately it is NOT possible to model this in a ManyToMany in Transfer - you can only sort by a property in the actual linked tables. So how do I achieve a sort based upon the 'SortOrder' field in the lookup table?

Well the solution is to forget the ManyToMany relationship and instead use a OneToMany in the user object that links to the lookup table which is defined as it's own object using composite keys. We then use a ManyToOne relationship from the lookup table object to the page object. The complete relationship definitions are as follows (including the user object):

<package name="users">

     <object name="user" table="tblUsers" decorator="model.users.user">
          <id name="id" type="numeric" />
          <property name="userName" type="string" column="userName" />
          <property name="password" type="string" column="password" />
               
          <!-- Link between a user and the pages they have access to -->
          <onetomany name="Pages" lazy="true">
               <link to="users.userPages" column="lnkIDUser"/>
                    <collection type="array">
                         <order property="sortOrder" order="asc"/>
                   </collection>
          </onetomany>
     </object>

     <object name="userPages" table="lnkUsers_Pages">               
          <compositeid>
               <property name="lnkIDUser"/>
               <property name="lnkIDContent"/>
          </compositeid>           
               
         <property name="lnkIDUser" type="numeric" column="lnkIDUser" />
         <property name="lnkIDContent" type="numeric" column="lnkIDContent" />
         <property name="sortOrder" type="numeric" column="SortOrder" />
               
         <!-- Link between a user and a content page -->
         <manytoone name="Page" lazy="false">
              <link to="content.content" column="lnkIDContent"/>
         </manytoone>

     </object>
</package>

<package name="content">
     <object name="content" table="tblContent" decorator="model.content.content">
     <id name="id" type="numeric" />
     <property name="title" type="string" column="title" />
     <property name="content" type="string" column="content" />
</package>

By using this relationship I am able to sort the pages collection by the 'SortOrder' field in the lookup table. This is possible as the lookup table itself is now an object with defined properties that we can access via the OneToMany relationship in the users object.

The only thing to be aware of with this setup is that the array you get back from getPagesArray() is composed of 'userPages' objects. You therefore need to call getPage() on each object in the array to access the page. E.G:

     <cfset userPages = getUser().getPagesArray()>
     <cfloop from="1" to="#ArrayLen(userPages)#" index="x">
          <cfset page = userPages[x].getPage()>
          <cfoutput>#page.getContent()#</cfoutput>
     </cfloop>

This is a flexible way to achieve a ManyToMany type relationship without having to use a ManyToMany composition. It allows you to add any fields you need to the lookup table and have access to them via a business object as well as being able to sort on any field in that table.

Spellify for Coldfusion V1.0 Released

Spellify is an awesome textarea spell checker which uses a smooth and effective Ajax based interface. It's written by Nikola Kocic and can be downloaded here: http://www.spellify.com/

I was immediately impressed by the look and feel of this project which uses the Google spelling API but was dissapointed to see that it only supported PHP and .NET.

Well, I couldn't have this so set to work on a Coldfusion version. As CF makes these kinds of thing simple, within 30 minutes I had knocked together a Coldfusion version of Spellify.php which makes the call to the Google API and returns the XML to the Spellify Javascript.

I haven't tested this massively but it's a simple piece of code and seems to work nicely so I thought I'd get it released so you can get integrating Spellify if you need a spellchecker for textareas on your site.

Let me know if you find any issues with it.

This *should* work with all versions of CF - at least back to CF6 anyway.

Download Spellify.cfm V1.0

Here are the instructions - taken from the included READ ME.txt:

======================================================================
Spellify - Spellify.cfm v1.0 (based on Spellify.php by Nikola Kocic - http://www.spellify.com/)
Copyright (c) 2009 James Allen. (jamesallen.name, www.jwadevelopments.com)

E-Mail: james@jamesallen.name
======================================================================

Spellify.cfm is a Coldfusion version of Spellify.php for use with the superb Spellify text area spellchecker.

INSTALLATION (the following is based on the default spellify.php install and using demo.html to test):

1. Download and unpack the Spellify PHP package: http://www.spellify.com/download.html

2. Place it somewhere in your webroot, e.g /wwwroot/spell/

3. Copy Spellify.cfm to the same location as Spellify.php in the Spellify sub-folder (e.g wwwroot/spell/spellify/spellify.cfm).

4. Edit 'spellify/src/spellify.js', line 47. Change:

var defUrl  = 'spellify/spellify.php';

to

var defUrl  = 'spellify/spellify.cfm';

That's it!

Spellify should now be working with Coldfusion. You will probably have to change the defURL when you implement the code on your site (e.g /functions/spellify/spellify.cfm etc).

Problems connecting to SagePay's TEST and SIM servers - FIXED

Since Tuesday afternoon we have found that we can't connect to the TEST or SIM servers at SagePay. Live is fine however, even though the exact same code is used.

It's a strange problem as an immediate connection failure is received when attempting to POST a transaction registration to either the test or sim servers.

After this had happened we contacted SagePay support who did some checks but told us that there are no problems on the server. However, when I did a quick Twitter search I found another developer who is also getting this issue. I confirmed the problem via my local development server in the office and on the live server so it doesn't appear to be a specific ISP routing problem etc.

However over the past few day SagePay have confirmed that a small number of user's are having problems. It's most certainy not affecting the majority though.

This has led support to believe that the problem is due to Coldfusion and the way it is sending the POST request to the SagePay server.

The Solution

Luckily while throwing this problem out to Twitter a very helpful user (@Zefer) suggested that the problem might be due to the server sending back a gzipped response which cfhttp in Coldfusion can't handle (and certain URL functions in PHP it seems). To fix this we just need to send headers to tell the server what we can accept:

<cfhttpparam type="Header" name="Accept-Encoding" value="deflate;q=0">
<cfhttpparam type="Header" name="TE" value="deflate;q=0">

The raw headers sent in the request will look like this:

Accept-Encoding: deflate;q=0
TE: deflate;q=0

Once I made this change the problem went away immediately.

I assume that SagePay must have made a system change on SIM and TEST but not on LIVE (luckily).

How to Secure Railo 3.1 Admin in IIS 6

I recently setup my new VPS with the awesome Railo 3.1 on Server 2003 and I soon started thinking about the security of the admin URL. With Adobe Coldfusion it's good practice to either move the administrator so it only runs on a non-standard port with IP filtering or protect it with a username and password via web server security.

I decided that I would go the route of securing the Railo admin path via IIS directory security, however I soon realised that in Railo the admin URL is not a physical path. I.E. It is not created as either a virtual directory or a real directory within the webroot of each website hosted on the server.

The Railo admin URL is as follows:

http://www.mysite.com/railo-context/admin/index.cfm

The way this URL is processed is that the ISAPI filter which Railo uses looks out for any call to railo-context and forwards it to the correct place within the Railo system. This is all done outside of IIS.

So how do we protect the URL?

The answer is actually pretty simple although you will have to do it on every virtual site setup on the server.

All we have to do is to create two physical folders within the webroot of the website we want to protect as follows:

railo-context/admin

Once this is created we can then go into the IIS 6 manager, locate the 'admin' folder, right click and select 'properties'. Within 'Directory Security' it is then possible to either restrict accces via IP or via authentication and access control.

If using username and password security go into the 'Authentication Methods' panel and uncheck 'Enable anonymous access' and leave 'Integrated Windows authentication' ticked.

Now, whenever you access the railo admin URL you can log in via an account on the server (e.g administrator). It's recommended that you create a special windows account that will only be used for admin access. You could then only give that user permissions to access the '/railo-context/admin' - though this would need to be done on each folder in each web root. In a hosted scenario this would be a good idea anyway as each customer should really have their own web user account to ensure they can only access their resources on the system.

Note: If you do decide to just log in using the administrator account on the server I recommend you do this over SSL ONLY. If you do it over http:// it would be possible for a user on the same network (e.g wifi access point) to sniff your http packets and determine your admin password - not good.

 

Monkey Island: Special Edition Released

Just a quick non-Coldfusion related post about the release of the classic point and click adventure game by LucasArts - Monkey Island.

Nineteen years after release the guys at LucasArts have taken the original game - keeping the exact same engine - and given it a visual and audiotory overhaul. They have redone all of the graphics (painstakingly redrawn by hand), updated the musical score and added a complete vocal track - using the voice actor from later Monkey games to play Guybrush again.

The results are frankly mind blowing. The graphics look fantastic and really give new life to this amazing example of the point and click adventure genre. I always longed to have Monkey Island as a 'talkie' but never thought it would happen. With this update I finally get to hear what all these awesome characters sound like and the results are brilliant.

I have only played it for a few minutes but got a huge buzz as soon as the opening titles appeared in glorious HD. I will be playing it on my 50inch plasma at 1080p resolution (1920x1080) and have to say it looks stunning.

It's being sold through Steam for the bargain price of £6.99 in the UK and $9.99 in the US.

It's a meaty 2.5gb or so download but the whole process is made nice and simple via Steam - I'm very impressed with the whole delivery mechanism I must say.

The game is available on PC via Steam direct download or on the XBox 360 via the Xbox Market Place,

You can get the PC version from here: http://store.steampowered.com/app/32360/

Details about the Xbox 360 version here: http://www.xbox.com/en-US/games/s/secretofmonkeyislandxbla/

Being able to play this again brings back some awesome memories of the first day I got my Amiga A500+ in 1990. Once I started playing I couldn't stop until it was completed.

I now just hope that it sells in bucket loads and convinces LucasArts to give the same treatment to Monkey Island 2: Le Chucks Revenge.

Here's some shot's of the game on my TV: (click for a larger view)


UPDATE:
You might not have realised yet but if you press F10 (on the PC version) at any time the game will revert back to the original look exactly - complete with actions panel at the bottom of the screen. It also removes the vocal track and goes back to the original music. Awesome to see the difference.

SagePay: Inconsistency in IFrame 'Low profile' option on Server solution - ** Workaround **

I've recently been integrating the SagePay Server solution into a client site and it was decided to utilise the new 'Low Profile' option (introduced in protocol 2.23). This new option allows the SagePay payment pages to appear within an Iframe placed inside the main site template. The obvious benefit of this is that the customer is not redirected to another site to complete payment and confidence in the vendor is improved.

The integration went pretty well with the SagePay XSLT templating system being relatively easy to use and customise. I soon had a template which nicely matched the style of the vendor's website.

During testing everything worked great. The payment form appeared within the IFrame, the card details could be entered and the 3D Secure page worked as expected. After the payment side had been handled SagePay redirect back to the vendor website outside of the IFrame - just what we need.

However, during testing I tested an AMEX card. AMEX cards are not part of the 3D Secure system and so the customer is not shown a 3D Secure page - just the card input form. Now this shouldn't really be a problem but when the card details have been entered and validated, SagePay redirect the customer back to the vendor site but inside the IFrame!

This is definetely NOT what we want as once redirected to our payment processing template the full site design is shown within the IFrame (header, sidebar and footer).

At this point I hit the HTTP debugger to find out what was causing this inconsistency. Here is what I found:

When 3DS authentication is required, the following HTTP requests are triggered:

https://test.sagepay.com/gateway/service/carddetails                302         POST     test.sagepay.com                /gateway/service/carddetails   
https://test.sagepay.com/gateway/service/cardconfirmation    302         GET        test.sagepay.com                /gateway/service/cardconfirmation       
https://test.sagepay.com/gateway/service/authentication         200         GET        test.sagepay.com                /gateway/service/authentication            
https://test.sagepay.com/mpitools/accesscontroler?action=pareq         200         POST     test.sagepay.com                /mpitools/accesscontroler?action=pareq                            
https://test.sagepay.com/mpitools/accesscontroler?action=auth            200         POST     test.sagepay.com                /mpitools/accesscontroler?action=auth               
https://test.sagepay.com/gateway/service/authentication?action=callback        200         POST     test.sagepay.com                /gateway/service/authentication?action=callback           
https://test.sagepay.com/gateway/service/authentication?action=completion                302         POST                test.sagepay.com            /gateway/service/authentication?action=completion
REDIRECT
http://dev.jamesallen.name/index.cfm/page/orders.order.cfm             200         GET                dev.jamesallen.name /index.cfm/page/orders.order.cfm

In this case, the final SagePay request is a POST (assumng it uses target="_top" which jumps the customer out of the Iframe)
The redirect back to the site is thus outside of the Iframe and all is good.


If 3DS authentication is NOT required the following HTTP requests are triggered:

https://test.sagepay.com/gateway/service/cardselection?vpstxid={283FE985-CE13-A510-B9E6-840531*****} 302                GET        test.sagepay.com            /gateway/service/cardselection?vpstxid={283FE985-CE13-A510-B9E6-840531*****}
https://test.sagepay.com/gateway/service/carddetails                200         GET        test.sagepay.com                /gateway/service/carddetails   
https://test.sagepay.com/gateway/service/carddetails                302         POST     test.sagepay.com                /gateway/service/carddetails   
https://test.sagepay.com/gateway/service/cardconfirmation    302         GET        test.sagepay.com                /gateway/service/cardconfirmation       
https://test.sagepay.com/gateway/service/authentication         302         GET        test.sagepay.com                /gateway/service/authentication
REDIRECT            
http://dev.jamesallen.name/index.cfm/page/orders.order.cfm             200         GET                dev.jamesallen.name/index.cfm/page/orders.order.cfm

In this case the last request inside SagePay is a GET which means the customer is left in the Iframe.

So there is a fundamental inconsistency in the implementation of the new 'Low Profile' option in the SagePay Server product. So, how do we solve this?

After some head scratching I came up with a solution which I feel is fairly straight forward and will ensure that when SagePay fix the problem, the site will not suddenly change it's behaviour and will be easy to modify to remove the Javascript redirect.

The solution is as follows:

  1. In your notification template (the one that SagePay POSTS back to and you respond with Status, RedirectURL, StatusDetail), set the RedirectURL to a new handler template (e.g http://site.com/orders/go.cfm). Then set a session variable to the URL you want the customer redirected to to continue the payment process.

  2. Create the handler template. This will build a dynamic form whose target is set to the redirect URL set in session in the step above. Javascript will be used to automatically submit the form when the page is loaded. An HTML submit button will be created inside the form in case the user does not have Javascript enabled.

That's the solution in a nutshell. We are modifying our notification template to always redirect to a new handler template which will jump the customer out of the Iframe and direct them to the correct template that continues the order process. The form target on the handler template is set to "_top" which causes the form submission to break out of the IFrame.

To help with this below is the code I use on my handler template (go.cfm). I use JQuery for the automatic form submission and to hide the submit button. You could of course code that functionality without JQuery.

<!--- URL to redirect to (set in session before the jump to here) --->
<cfset targetURL = session.targetURL>

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN"><html>

<head>
    <meta http-equiv="Content-Type" content="text/html;charset=utf-8" />
    <link rel="stylesheet" href="/styles/styles.css" type="text/css" />
   
    <title>Payment Processor</title>
   
    <script language="javascript" src="/functions/jquery.js"></script>
    
    <script language="javascript">
        $(document).ready(function() {
            $("#jump").hide();
            $("#jump").submit();
        });
    </script>
</head>

<body>
    <cfoutput>
        <form id="jump" action="#targetURL#" target="_top">
            <input type="submit" value="Click to continue your order">
        </form>   
    </cfoutput>
</body>
</html>

With this approach, once SagePay fix the inconsitency I can simply change this template to perform a server redirect to the required URL instead of using the Javascript and self submitting form.
Either that, or to change the notification template so that it goes directly to the required URL's as before.

Hopefully this will help some people who have got this problem when implementing the Server solution.

More Entries

© 2014 James Allen | Contact Me
This blog runs on the awesome power of BlogCFC - created by Raymond Camden. This blog is running version 5.9.