Thursday, December 24, 2009
Parsing simple, tabular, flat files like the one below does not really pose a large challenge as either the OS (Text Drivers) or CF (CFHTTP) have tools to read them in without much fanfare.
Example 1: Tabular Fixed File
However, many times over, the examples are not as simple. The data on each row may be different; relationships may exist between lines of data etc.
Example 2: Complex Fixed File
For the complex files we normally write specialized parsers or transformations. Certainly a whole industry exists that deals with Extract Transform Load processes. Either choose requires considerable programming efforts.
After writing a parser for the umpteens fixed file that someone wanted to have loaded, I thought there needs to be a better way. The short of all this is that I created a more generic component to deal with this in native ColdFusion. The solution was to split up the code and the file definition.
Thus, in order to parse complex files, you provide an XML definition file which describes the flat file. Then call the component to do the heavy lifting.
Example 3: Flat File Definition XML
The outcome of such processing are standard ColdFusion objects that are easier to deal with and require less programmatic effort to implement.
I posted this project on RIAForge (FixedFileReader). It contains examples for different scenarios of varying complexity. I added skeletons for EDI X.12 and VCF4 formats.
May not work for all, but if it does, it is likely that it will save a lot of time.
Wednesday, November 18, 2009
You, then, use a CFTRY/CFCATCH combination to handle timeout occurrences, e.g. a web page failed to respond, or FTP didn't connect, etc.
However, this system breaks down if your intent is control multiple tags in combination, e.g. a combination of CFHTTP and CFFTP may take up to 30 seconds though individually CFHTTP or CFFTP may consume a variable amount of time.
Similarly, you are running a multitude of small steps in a loop, e.g. loading 1000 records, or displaying 500 lines. Though each line will only take a fraction of a second to process the number of lines in aggregate will blow you out of the water.
Normally in these cases we would wait on an overall page time out if we had one, then produce a generic catch all message saying your page has timed out and good luck trying this again.
This may suffice generically speaking, but may prove to be frustrating for users and visitors of your apps and sites.
What if you could instead be more pro-active. What if you could predict how long your processing is going to take, even on shared servers with varying loads?
What if you could give users options to either proceed, adjust your number of iterations, or simply stop early because you will never finish the task at hand in the time given.
Ideally your code would be very compact:
<!--- check prediction --->
<!--- we will timeout lets do something about it --->
Conceptually speaking this could have been done for a long time in CF, but practically I have seen few implementations.
Let's look at what it would take to make this prediction using a scenario in which we have a repetitive core process that we need to time. Thus, it would be all about the ability to measure time for each loop. Thus we would
a) need to start a timer, e.g. GetTickCount()
b) end the timer, e.g. another GetTickCount()
c) measure the difference between start and end time
d) use the remaining iterations to come up with how much more time we need
Simple, right? Indeed it is. But the downfall is that it doesn't work.
The system breaks down because of the high variability in practice between each iteration of the loop. Using the built in CF function GetTickCount() we could establish a start and end time. But, when we measure the execution time between the two tick counts the results in this realm change quite a bit.
So, will we have to implement something complex using Java JETM instead to get this working? You probably could, but it isn't necessary if we add a dose of statistics to the whole puzzle. Here is a modified approach:
a) establish statistical validity threshold in number of iterations (e.g. how many time do I need to have gone through the loop to have a valid sample)
b) still need to start a timer, e.g. GetTickCount()
c) end the timer, e.g. another GetTickCount()
d) measure the difference between start and end time
e) update your statistics
f) if validity threshold has been reached make prediction
This does seem to work a lot better but requires many more lines of code. Rather than put all of these lines of code into each template, I have abstracted the needed calls into a component.
I would initialize the component (objTC) with the number of iterations (number of times I would need to go through the loop) and how many times I will need to have completed the loop to get a valid time prediction. In my example this would be 1000000 iterations with 150 completions. Or, something like this:
<cfset objTC = CreateObject("component","TimeControl").init(1000000,150)>
Then within the loop I would start and end the main timer like so:
<!--- start measuring --->
<!--- end measuring --->
Now that the component takes care of all the nasty statistics stuff I can keep my main code simple and have some functions to use to either provide feedback to user, or just check on the status of things.
E.g. I could provide feedback like so:
<!--- send the expected time to user
when this process will complete --->
Expected completion at #objTC.getFinishTime()#
or (#objTC.getFinishSeconds()#seconds) <br/>
This is just a sample on how to handle it. I have put several more functions in the examlpe to play with. It may work well for some of your scenarios or not all. In any case, I do believe an improved feedback to user and more predictive code behavior based on execution time will reflect well on you as programmer. So feel free to play with samples and see whether this can get you started.
The full sample code can be downloaded here.
Saturday, November 14, 2009
Also we discussed with special focus on time based debugging from the performance perspective as well as from the perspective of handling time based exceptions predictevly.
Dowload presentation and code now.
Several questions regarding the use of CFDUMP came up and here are couple of links to deeper treatment of advanced CDFDUMP tag in CF9. Additinal attributes can be specified to tweak its behavior:
CFDUMP in CF9
Wednesday, October 21, 2009
Download Beginning Debugging Session
Download Advanced Debugging Session
I have inlcluded more slides and examples than we covered hoping that they can be usefull as well. All my material are distributed under creative commons license. Feel free to contact me with questions or comments.
Generally, this conference showed that the spirit of CF is alive and well. It was an all volunteer event from the organization to the speakers and all the things in between. It introduced old and new people to CF and furthered networking in this Region. I hope we can make this an annual event.
Tuesday, October 13, 2009
Debugging is the often hated but yet nonetheless very important part of the developer's life fraught with lots of frustration and brief moments of glorious joy when the culprits are found. I am trying to give both the beginner and advanced audiences multiple techniques and tools to make the shining moments more frequent and frustrating times less so.
Here is my official description about the overall presentations:
Debugging can be a very frustrating exercise. Many times we find ourselves spending three times as long as it took us to develop the code to debug it. Especially code written by other developers seems time consuming to debug ;o)
Understanding the debugging infrastructure will shortcut much of the confusion and help to quickly focus on the source of the problem. What can you do when
Hope to see you there,
Thursday, September 24, 2009
I quickly chatted up the Adobe people at the conference on how to accomplish this as I did have a project requiring deployment of 23 CF servers. They all were consistent, which is good; I need to talk to the Eastern Sales Rep (Dominick Conte) whom I could not find in person or in attendance anywhere, which is bad, but whose business cards where everywhere ;o)
So, upon my return, I happily attempted to call Dominick. Different times, different dates; only voice mail answering. I thought the guy must be selling this stuff like hot-cakes as he never is available via phone; so I gave up and send an email to ask how one orders these hot deals offered by Adobe on the show.
His initial response was that I was not really at the show. Interesting, approach; made me doubt my very existence for about a little less than 0.1 ms... ahhh... deep thoughts. Of course, I advised him how he can find out who attended and then was waiting for some sort of contact.
Alas, again, no-go.
I am now paraphrasing the emails that went back and forth thereafter:
Me: "hey wassup", "why you not talk to me?", and "how can I order your good stuff?" and "why is it so difficult to talk to you guys?".
Dom: "Yeah, I tried to talk to you" (I checked and no such phone logs), and " what the h..k do you want anyways?"
This is not going well I think, as I am very consitently asking this man who is in sales on how to place an order (partners, direct or what?). So, he finaly caves and shoots back an email that he is not really the dude to talk to and the only way I can get this offer is to talk to a company in Canada.
We finally did end up buying some licenses for different projects from Adobe through the reseller in Canada. The credit for this goes entirely to Adam Lehman who could get me a contact name for ordering.
Wednesday, September 2, 2009
Here are the details on the conference:
In order to keep the event free we have removed some hard costs and some benefits you normally get from paid-registration events.
Please be aware of the following:
1) You are responsible for paying for lunch for both days. We have enlisted the services of a local caterer and will be providing lunch each day for $10/day. This includes a sandwich, chips and drink. Please bring cash the day of the event! We will not be able to process credit cards!
2) Limited Edition, Collectible CFinNC conference t-shirts will be available for purchase for $15 with any proceeds going to offset costs and possibly sponsor a planning committee dinner (if we sell them all).
Lunch and shirt purchase is completely optional. You may indicate if you would like to purchase lunch and/or a t-shirt on the registration
For more information, please check the CFinNC website at: http://www.cfinnc.com
Wednesday, August 19, 2009
So far the hotfixes did not appear to break anything but the packaging is lacking as many manual steps will have to be completed.
I could not quite understand why these hotfixes could not have been bundled together as one fix that can be applied.
To make our lives easier I have built an installer for ColdFusion 8.0.1 running on Windows systems to do just that. You are free to use it at your own risk.
The installer will update standalone installation of CF not J2EE/JEE installations.
Thus the following hot fixes will be applied:
CVE-2009-1872, CVE-2009-1875, CVE-2009-1876, CVE-2009-1877, CVE-2009-1878
These JRUN only updates will not:
- Open source coldfusion is engaged and attempting to move things forward. Adobe is cooperating for now. Railo was there and willing; OpenBD was there in spirit.
- ColdFusion builder is here despite the CFEclipse and we can expect more opinions on why one is better than the other. Adobe did not want to play in this open source game.
- ColdFusion 9 will have some expanded licensing options to make it easier to host things in the cloud.
- ColdFusion has gained some momentum but still overall a nishe.
- The food was pretty good.
- Presenters could focus more on content and less on opinion.
- Some are born to present; others not so.
- ColdFusion is still niche and we need to not get so high on ourselves.
- Flex purist insistance that they do not need to know anything about CF.
- No CFUnited backpacks for the masses. What is up with that?
- One hour barely covers anything technical well. More multi-hour tracks digging into topics would be helpfull for people seeking pure technology how to.
Wednesday, July 29, 2009
Also to note is that ColdFusion has supported a mechanism to serialize complex data via WDDX for a long time (I believe since version 4).
However, in my case, the need was not for serialization components but rather for compact serialization. In other words, use as little space (bytes/data) as possible. In addition, I needed to easily save and retrieve this information from a database in a text based format. Oh, yes, and handle complex data objects such as structures and arrays.
WDDX, though usable, is very verbose and thus was out. Looking at the Java API and reading through posts I translated this, in the end, to two functions fSerialize and fDeserialize. You will need ColdFusion 8.0.1 or higher to make this work.
The fSerialize function:
<cffunction name="fSerialize" access="public" returntype="string"
hint="uses java byte streams and Base64 encoding to serialize CF objects, this can be used instead of WDDX tag CFML2WDDX">
<cfargument name="input" type="any" required="Yes" hint="the CF object to be serialized">
var objByteStream = createObject("java", "java.io.ByteArrayOutputStream").init();
var objOutStream = createObject("java", "java.io.ObjectOutputStream").init(objByteStream);
var objSerialized ="";
//turn CF object in argument to out stream
//take outstream and make bytearray
objSerialized = objByteStream.toByteArray();
//encoded it and return
<cffunction name="fDeserialize" access="public" returntype="any"
hint="uses Base64 encoded Java Byte Array and turns to CF object. This can be used instead of WDDX WDDX2CFML">
<cfargument name="Input" type="string" required="Yes" hint="the Base64 encoded ByteArray that used to be a CF object to be deserialized">
var objSerialized =BinaryDecode(Arguments.input,"Base64");
var inByteStream = createObject("java", "java.io.ByteArrayInputStream").init(objSerialized);
var objInStream = createObject("java", "java.io.ObjectInputStream").init(inByteStream);
var objCF = objInStream.readObject();
//return the read object
These function on average were using 50% of the storage that a comparable serialized WDDX object would, so they achieved their objective for me. They will work on small components (cfc) but I have not tried to serialize very complex cfcs.
Wednesday, May 20, 2009
I, being a renegade, wanted to use it to do Database Load testing. There are great new facilities in VSTS 2008 to create Database Unit tests but none speaks of how to do a load test.
I thought it would be easy enough to adapt the techniques in web-testing to database load testing. I found out I was very naive. I should have caught on when:
a) There was no documentation at all, no video, no web post, no nothing from MS or anyone having done this.
b) The number of errors I was getting when even trying to run it with one agent.
My test setup initially:
a) One workstation running controller and VSTS 2008
b) One server running VSTS Load Agent
c) A beefy SQL Server 2008 box to be load tested
Load testing on my workstation was working great, everthing was as expected and I was hopefull that this could easily translate into working well on the agent system. In short, it did not:
Here is the initial error I got:
- System.IO.FileNotFoundException: Could not load file or assembly
I changed the test rig in this fashion:
a) Workstation with VSTS 2008 and controller + SP1
b) Server with VSTS 2008 and Load Agent + SP1
c) SQL Server 2008
The solution, thus, is to install visual studio team system 2008 on the agent computer/server as well, then install the agent. What good is an agent that does not cover all test cases? That is something that Microsoft hopefully will resolve. There seem to be dependencies on VSTS core files, I played with it a little. You do not need:
a) Any Crystal stuff
b) SQL Server Express
d) dotfuscator stuff
e) Documentation or redistributables
My test where in C# so I installed on support for it and no other language.
Wednesday, April 29, 2009
I have this little Java project which requires that I connect to SQL Server 2005. Something I have done many times. Download the JDBC drivers from MS and copy them into the lib directory and do your shindig in the connection to get going.
But, this time things have changed and the installation instructions are worthless at best.
Here is what I did:
- Downloaded the driver files from MS for JDBC 2.0
- Unzipped to directory
- Moved jar files (there are two files sqljdbc.jar and sqljdbc4.jar) to my lib directory
- coded my connector to use Class.forName("com.microsoft.sqlserver.jdbc.SQLServerDriver")
- Did some query code
And presto, nothing works as expected. Instead I get an exception with this in the stack trace:
"The JDBC Driver version 2.0 does not support JRE 1.4. You must upgrade JRE 1.4 to JRE 5.0 or later when using the JDBC Driver version 2.0. In some cases, you might need to recompile your application because it might not be compatible with JDK 5.0 or later. For more information, see the documentation on Sun Microsystems Web site"
Complete bogusness as neither Microsoft or Sun contain much info about how to solve this. So, head scratching starts followed by experimentation.
Come to find out I made a mistake by copying both jar files packaged in the driver to lib. They contain similar classes and the sqljdbc.jar files classes' are used if both are copied. However, the sqljdbc.jar file classes are not ment to be used with jre/jvm 1.6+ . Why noone mention this in the provided install doc is a different question.
a) Remove the not needed jar file (sqljdbc.jar) from lib directory or
b) specifically declare to use only the right one (sqljdbc4.jar)
Microsoft, please help us measly developers wasting time and state this in basic terms in your install doc.
There you have it.
Saturday, March 28, 2009
Say you have several functions in components processing away happily, and you want to give the user some feedback to keep him/her entertained as well as keep their paws of the back buttons etc..
You think doing couple of flushes during your processing may be a good way until you try and get this error:
Unable to perform cfflush.
|You have called cfflush in an invalid location, such as inside a cfquery or cfthread or between a CFML custom tag start and end tag.|
What to do? Since none of the hints apply to you. Your code looks like this:
<cffunction name="fFlushIt" hint="flushes current content">
<cfargument name="feedback" default="">
<cffunction name="fProcessing" returntype="numeric"
hint="does processing" output="No">
<cfset var x=10>
<cfset var y=20>
<cfset fFlushIt("Processing Complete")>
<cfreturn x + y>
<cfoutput>The result is: #fProcessing()#</cfoutput>
here comes the digging into the code and making guesses part. It seems that there is one more scenario cfflush won't like and this is if you call in within a call-tree (yes even nested stuff) in a function that has the output attribute declared as "No". Maybe a hint like that could be placed in the error to avoid all that brain scratching and wondering that goes along with this?
Anyways change the function like this and it worked. Yeah !
<cffunction name="fProcessing" returntype="numeric"
hint="does processing" output="Yes">
Thursday, March 26, 2009
Thousands of them. Ran the scripts into Oracle using ColdFusion instead of SQLPlus only to find out that I now had thousands of invalid triggers.
A little puzzling, as the same scripts worked like a charm everywhere else. I go use the oracle web enterprise manager to see whehter I can recompile them and make a few valid. Nada!
Everytime the Enterrpise manager tries to compile the trigger I get the error:
PLS-00103: Encountered the symbol ""
Here is a sample trigger with issues:
CREATE OR REPLACE TRIGGER INS_MySuperTrigger BEFORE INSERT ON
MySuperTriggerTable REFERENCING OLD AS OLD NEW AS NEW FOR EACH ROW
If :new.MySuperTrigger_ID IS NULL THEN
SELECT SEQMySuperTrigger.NextVal INTO newid FROM dual;
:new.MySuperTrigger_ID := newid;
This is nuts. I use the generate SQL button of Enterprise manager to generate the SQL and copy and paste it into the Oracel SQL Developer UI, run the code without mods, and bingo; the trigger is valid and happy as a peach.
What gives? Long hours wasted with different websites and options, casing, single and double quotes, Egyptian prayer beads..., you name it, I tried it.
Until pulling a protocol sniffer to see what the difference is between Oracle Enterprise Manager and Oracle SQL Developer on the wire.
Come to find out Oracle does not recognize Windows CRLF (Chr(13) + Chr(10)) as blank space, if you replace all the CRLF with LF this works like a charm. Seemingly, web based Oracle Enterprise Manager does not do this translation, while the Oracle SQL Developer tool does. Yack! Lesson learned, I now run the scripts through a parser before running them to Oracle via JDBC and get valid triggers all the time.
Tuesday, March 24, 2009
When using the command line mysql tool you can specify the username and password of the user during login.
Many manuals describe the syntax as:
mysql -h hostname -u root -p password
mysql --user=user_name --password=your_password db_name
You think wow this is easy, right. I can interpret this very simply.
I go to the command prompt and type:
C:\>mysql --user=root --pasword=mypass
but to my astonishment I would get:
mysql: unknown variable 'pasword=mypass'
Here are couple of other beauties that are in the manuals and don't work at all like expected:
C:\>mysql -u root -p mypass
Enter password: ******
ERROR 1045 (28000): Access denied for user 'root'@'localhost' (using password: YES)
This could be very frustrating to someone new to MySQL on windows. Can't even login to get the client program started.
The approach I found working for windows based installation is to ignore the manual and avoid space between parameters.
Thus rather than using -u root, you will need to use -uroot. This will magically work. E.g.:
C:\>mysql -uroot -pmypass
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 19Server version: 5.0.67-community-nt
MySQL Community Edition (GPL)
Type 'help;' or '\h' for help. Type '\c' to clear the buffer.
Hope this will avoid some frustration out there.
Thursday, March 19, 2009
Broke out the old Hex editor to save and check file contents. Output each character at a time, just could not see why this would happen.
Many hours later, a headache started to kick in. Time for a break. Then, a hunch started knocking at the back of my scull and got louder and louder.
To prove it to myself, I created an intermediate variable into which I stored the function result. When this scheme was used, no space. If I used the function output in place, space! It turned out to be the Output attribute of the function definition. You will need to set it to 'No'. If you leave it off and attempt to use the function in place, Coldfusion will introduce a space in the return. Very, very annoying.
Here is sample code to reproduce the problem, two function that are identical except one has the output attribute specified; if you use the function 2 inline CF will produce a space in front of the output; see the output for f2 below:
<cffunction name="fFunction1" output="No">
<cfset var strReturn ="Hello World">
<cffunction name="fFunction2" >
<cfset var strReturn ="Hello World">
f2:-- Hello World--
Monday, March 9, 2009
To CF it looked like all of a sudden the content of the HTTP traffic vanished, in other words we did not seem to have anything in GetHttpRequestData().content except for empty string.
Of course, no one fessed up to having changed anything at all, so here I go digging through code and cannot find anything obvious and, then, I turn out the big guns: Protocol Sniffers and packet capture.
Hah, I think, now I can show them that they are not sending any content, but instead I can see the content clearly in the packet capture which starts heavy head scratching and coffee sipping.
More looking into code and I find that the only time we bypass GetHttpRequestData().content is if we are running a binary check, e.g. we have this condition:
From the protocol packets I can clearly see that the content is not binary so, at first, I cannot imagine how this condition would be even trigger and bypass the remainder of the processing until, seing the light, I remove the condition.
Bingo, now I can see content but it is all wrong. It looks like CF has processed the content of the transmission to numeric values instead of XML strings.
So in the end, more digging to see that the http content type declaration of the originating transmission had changed from text/xml to application/octet-stream; it looks like whenever CF received this it automatically converted the perfectly fine XML into an octet stream, which, then, turned the content to be binary data instead of leaving it as simple string. A simple misdeclaration by the customer which caused this hoopla.
So after much searching I wish I could have been able to tell the RAW data in the transmission from the interpreted data. This would have made the diagnosis much simpler. So Adobe, in the future please consider adding the raw information to the GetHttpRequestData function, so we don’t have to guess what parts are being interpreted and what the transmission contained.
For now, I went back to the customer and asked them to correct their content declaration.
Sunday, February 22, 2009
Thus, the safe route to use is to go through the database to round anything, unless you really do not have any choice.
This time it was the LSCurrencyFormat function which caused the headaches. It will round down at the .5 fraction rather than round up which is very annoying and disturbing at the same time. Here is a function specifically made for currency handling and it does not handle the basic calculations correctly.
Example code (comparing the round behavior against LSCurrency):
<cfloop from="0.001" to="0.009" step="0.001" index="fraction">
<cfset Amount = 1.10 + fraction>
number: #Amount# ls: #LSCurrencyFormat(Amount,'none')#
compare to rounded #Round(Amount *100)/100# <BR>
Unfortunately, I had flip back all the use of LSCurrencyFormat and pre-round the numbers via the database before passing them to this function. I do hope that Adobe does a little more testing on rounding for these in the future.
Thursday, February 12, 2009
Well, at least if I could detect whether I am in a transaction I could write around this I think. But there is no way that I have found. No clear posting on how to do this.
First approach I used was to create a function that would open and close a transaction, then detect the error thrown. If the error was thrown I was assuming that we were in a transaction and thus could not open a new one, a fuction like this:
<CFSET var blnReturn = false>
<!--- emptry transaction tag --->
Unfortunatly, this does not work. When CF throws an error for nesting, even within the try/catch block for the purposfully nested transaction, the transaction wrapper is removed. Thus you are hosed if an error occurs later down the execution.
What to do then?
After much researching and failure, here is the approach I did find working. The trouble with this is, that there is no guarantee that it will work in future versions of CF, which I hope will introduce a simple function like InTransaction() . We are using the ColdFusion Java implementation of the Transaction Tag to find out whether we have a current transaction. This is the fully wrapped function.
<CFFUNCTION name="InTransaction" access="public" displayname="checks to see whether we are currently running a database transaction. returns true or false." output="No" RETURNTYPE="boolean">
<CFSET var objTrans ="">
<CFSET var blnActiveTransaction=false>
<CFSET var objCurrentTrans="">
<!--- Call to CF implementation of TransactionTag to expose Java functions --->
<cfobject type="Java" class="coldfusion.tagext.sql.TransactionTag" name="objTrans" action="create">
<!--- objCurrentTrans will become undefined if the Java function call getCurrent() returns Null,
otherwise this returns a pointer to current transaction instance --->
<cfset objCurrentTrans = objTrans.getCurrent()>
<!--- return result --->
This works in ColdFusion 8 and 7.