Let me start by saying that I like unit testing . It is very valuable tool in the arsenal of developers to fight the ever present monsters of recursion bugs and integration nightmares.
However, I will also freely admit that there is no agreement on how many unit tests are ever enough. Put a three developers in a room and you will get three different answers and maybe a headache to boot.
Also, in my case, the other challenge was to determine whether I had sufficient permutations of tests to verify that unit test goals could be met. Mind you that, test-coverage does not equal code coverage, but it probably is a good proxy. So, another goal of mine is to be able to think through all kinds of ways to call on code even the bad stuff and ensure that it behaves as expected. There is system in this, however, and this is what we use come up with most of the tests.
With MXUnit the world of ColdFusion has had a sturdy companion to write all these nice unit tests, however, we needed more. We wanted to have a good number of tests especially for cfc (ColdFusion components). Coming up with the variety and permutation of tests does require quite a bit of hand coding, so we were looking for an easier way.
The solution we came up with was to generate test stubs. Many, many, many test stubs. We then review the cases and expand the ones we think cover the objective sufficiently. We end up deleting quite a few generated test, but this gives us a good baseline, especially for things we don't commonly unit test but we should, e.g. sending in a number when a string is expected, sending complex values, when simple ones will do and vice versa.
This may not be the way you want to work all test cases but helps to take care of many. We look at a functions parameters and generate all combinations of parameters with standard and break values. This can amount to be many thousandths so use with care.
The generator sample code I am attaching has been squeezed into one code file (cfm template) so it is easier to post. Place the code content into a file named "GenerateTestsPublic.cfm" under your mxunit path. Not optimal but the concepts should be visible. Take it for a spin and make adjustments. Feel free to add comments to the blog post.
Happy Experimenting:
GenerateTestsPublic.cfm:
Cheers,
B.
However, I will also freely admit that there is no agreement on how many unit tests are ever enough. Put a three developers in a room and you will get three different answers and maybe a headache to boot.
Also, in my case, the other challenge was to determine whether I had sufficient permutations of tests to verify that unit test goals could be met. Mind you that, test-coverage does not equal code coverage, but it probably is a good proxy. So, another goal of mine is to be able to think through all kinds of ways to call on code even the bad stuff and ensure that it behaves as expected. There is system in this, however, and this is what we use come up with most of the tests.
With MXUnit the world of ColdFusion has had a sturdy companion to write all these nice unit tests, however, we needed more. We wanted to have a good number of tests especially for cfc (ColdFusion components). Coming up with the variety and permutation of tests does require quite a bit of hand coding, so we were looking for an easier way.
The solution we came up with was to generate test stubs. Many, many, many test stubs. We then review the cases and expand the ones we think cover the objective sufficiently. We end up deleting quite a few generated test, but this gives us a good baseline, especially for things we don't commonly unit test but we should, e.g. sending in a number when a string is expected, sending complex values, when simple ones will do and vice versa.
This may not be the way you want to work all test cases but helps to take care of many. We look at a functions parameters and generate all combinations of parameters with standard and break values. This can amount to be many thousandths so use with care.
The generator sample code I am attaching has been squeezed into one code file (cfm template) so it is easier to post. Place the code content into a file named "GenerateTestsPublic.cfm" under your mxunit path. Not optimal but the concepts should be visible. Take it for a spin and make adjustments. Feel free to add comments to the blog post.
Happy Experimenting:
GenerateTestsPublic.cfm:
<!---
Generate mxUnit tests given a component name.
One File per component function will be generated in this directory.
Will attempt to generate all permutations possible.
Random values
Break Values
Prefix:
Test_[componentName]_[functionName].cfc
An overall TestSuite will be generated
TestSuite_[componentName].cfm
Distributed under Apache 2 Lincese
(c) 2012 Bilal Soylu
--->
<!DOCTYPE HTML>
<html>
<head>
<title>Generate MXUnit Test for Components</title>
</head>
<body>
<CFIF IsDefined("Form.objectName") AND Trim(Form.objectName) NEQ "">
<!--- get object info --->
<cftry>
<cfset objReg = CreateObject("COMPONENT","#Trim(Form.objectName)#")>
<cfset stcMeta = getMetaData(objReg)>
<!--- generate directory --->
<!--- generate test files --->
<!--- save object name --->
<cfscript>
//save main object
strName = UCase(Trim(Form.objectName));
strUseObjectNameInDir = ReplaceNoCase(strName,".","_","ALL");
strDirPrefix = "unitTests";
strHint="";
arrTestCaseNames = [];
//max test cases (high number of combinations can exist
intMaxCases = -1;
intMaxCasesPerFile = 1000;
if (IsDefined("Form.maxTests") and Val(Form.maxTests) GT 0) intMaxCases = Val(Form.maxTests);
if (IsDefined("Form.maxTestsPerFile") and Val(Form.maxTestsPerFile) LT intMaxCases) intMaxCasesPerFile = Val(Form.maxTestsPerFile);
//names & paths
strBasePath = GetDirectoryFromPath(GetCurrentTemplatePath()) & strDirPrefix & "\";
strFilePrefix = "Test_#strUseObjectNameInDir#_";
strTestSuiteName = "TestSuite_#strUseObjectNameInDir#.cfm";
strTestSuitePath = strBasePath & strTestSuiteName;
strTestCaseDirName = "TestCases_#strUseObjectNameInDir#";
strTestCaseDirPath = strBasePath & "" & strTestCaseDirName;
blnComplete=false;
crlf = chr(13) & chr(10);
tab = chr(9);
//create directory
if (NOT DirectoryExists(strTestCaseDirPath)) DirectoryCreate(strTestCaseDirPath);
//sample data
sampleStructure= ":" & SerializeJSON({"number"=9999,'text'='my text value','dte'=Now()});
sampleArray= ":" & SerializeJSON(["a","b",33,{"number"=9999,'text'='my text value','dte'=Now()}]);
sampleQuery = QueryNew("");
FastFoodArray = ["French Fries","Hot Dogs","Fried Clams","Thick Shakes"];
nColumnNumber = QueryAddColumn(sampleQuery, "FastFood", "VarChar", FastFoodArray);
</cfscript>
<!--- assemble files --->
<cfif IsDefined("stcMeta.Functions") AND ArrayLen(stcMeta.Functions) GT 0>
<cfloop index="i" from="1" to="#ArrayLen(stcMeta.Functions)#" step="1">
<!--- each functions has its on test case. init var containers --->
<cfset stcFunc = stcMeta.Functions[i]>
<cfset selParaCombinations = QueryNew("")>
<cfset stcQueries = {}>
<cfparam name="stcFunc.Access" default="public">
<cfif stcFunc.Access IS "public">
<cfset strMethod = UCase(stcFunc.name)>
<cfset strMethodHint = "">
<cfset strTestCaseName = strFilePrefix & strMethod>
<cfset strTestCaseFileName= strTestCaseDirPath & "\#strTestCaseName#.cfc">
<!--- output (bsoylu 03-29-2012) --->
<cfoutput>
<br>processing: #strMethod#<br>
</cfoutput>
<!--- start testCase tC variable --->
<cfset genStartTc()>
<!--- init --->
<cfset arrParams = stcFunc.Parameters>
<!--- iterate through each paramter and set base test values --->
<cfif (ArrayLen(arrParams) GT 0)>
<cfset strQList = "">
<cfloop from="1" to="#ArrayLen(arrParams)#" index="y">
<cfset stcPara = arrParams[y]>
<cfset strParaName= UCase(stcPara.Name)>
<!--- create query and query handle --->
<cfset stcQueries["q_#strParaName#"] = QueryNew(strParaName,"CF_SQL_VARCHAR")>
<cfset selQ = stcQueries["q_#strParaName#"]>
<cfset strQList = ListAppend(strQList,"q_#strParaName#")>
<!--- if parameter is not required it can be null as a valid state --->
<cfif NOT (IsDefined("stcPara.Required") AND stcPara.Required)>
<cfset QueryAddRow(selQ)>
<cfset QuerySetCell(selQ,strParaName,"NULL")>
</cfif>
<!--- by type --->
<!--- we prefix with equal sign if we need to eval the data later in processing --->
<!--- we prefix with colon (:) when we need to deserialize JSON --->
<cfif IsDefined("stcPara.Type")>
<cfif stcPara.Type IS "numeric">
<!--- for each numeric: use max, use min, use zero, use random --->
<!--- Java Long: 9223372036854775807 and -9223372036854775808 --->
<cfset QueryAddRow(selQ)>
<cfset QuerySetCell(selQ,strParaName,"999999999")>
<cfset QueryAddRow(selQ)>
<cfset QuerySetCell(selQ,strParaName,"-999999999")>
<cfset QueryAddRow(selQ)>
<cfset QuerySetCell(selQ,strParaName,"0")>
<cfset QueryAddRow(selQ)>
<cfset QuerySetCell(selQ,strParaName,"#RandRange(0,999999999)#")>
</cfif>
<cfif stcPara.Type IS "string">
<!--- for each string: use max, use empty, use "coldFusion" --->
<cfset strLongString = RepeatString("a",4000)>
<cfset QueryAddRow(selQ)>
<cfset QuerySetCell(selQ,strParaName,"#strLongString#")>
<cfset QueryAddRow(selQ)>
<cfset QuerySetCell(selQ,strParaName,"")>
<cfset QueryAddRow(selQ)>
<cfset QuerySetCell(selQ,strParaName,"coldFusion")>
</cfif>
<cfif stcPara.Type IS "struct" OR stcPara.Type IS "any" >
<!--- for each struct: use empty, use test struct --->
<cfset QueryAddRow(selQ)>
<cfset QuerySetCell(selQ,strParaName,sampleStructure)>
<cfset QueryAddRow(selQ)>
<cfset QuerySetCell(selQ,strParaName,"=StructNew()")>
</cfif>
<cfif stcPara.Type IS "date" >
<!--- for each date: use "1/1/1980" use today use tomorrow, use 12/31/2200 --->
<cfset QueryAddRow(selQ)>
<cfset QuerySetCell(selQ,strParaName,"=CreateDate(1980,1,1)")>
<cfset QueryAddRow(selQ)>
<cfset QuerySetCell(selQ,strParaName,"=Now()")>
<cfset tomorrow = DateAdd("d",1,Now())>
<cfset QueryAddRow(selQ)>
<cfset QuerySetCell(selQ,strParaName,"=CreateDate(#Year(tomorrow)#,#Month(tomorrow)#,#Day(tomorrow)#)")>
<cfset QueryAddRow(selQ)>
<cfset QuerySetCell(selQ,strParaName,"=CreateDate(2200,12,31)")>
</cfif>
<cfif stcPara.Type IS "array" >
<!--- for each array: add sample array and empty --->
<cfset QueryAddRow(selQ)>
<cfset QuerySetCell(selQ,strParaName,"=ArrayNew(1)")>
<cfset QueryAddRow(selQ)>
<cfset QuerySetCell(selQ,strParaName,sampleArray)>
</cfif>
<cfif stcPara.Type IS "boolean" >
<!--- for each bool: true/false --->
<cfset QueryAddRow(selQ)>
<cfset QuerySetCell(selQ,strParaName,"Yes")>
<cfset QueryAddRow(selQ)>
<cfset QuerySetCell(selQ,strParaName,"No")>
</cfif>
<cfif stcPara.Type IS "query" >
<!--- for each query: sample and empty --->
<cfset QueryAddRow(selQ)>
<cfset QuerySetCell(selQ,strParaName,":#SerializeJSON(sampleQuery)#")>
<cfset QueryAddRow(selQ)>
<cfset QuerySetCell(selQ,strParaName,":#SerializeJSON(QueryNew(""))#")>
</cfif>
<cfelse>
<!--- valid state for non-typed or unknown paramters --->
<!--- for each undefined type (any): use structure --->
<cfset QueryAddRow(selQ)>
<cfset QuerySetCell(selQ,strParaName,sampleStructure)>
</cfif>
<!--- for the first parameter query we also set the query result. If there is only one parameter this is what will be returned --->
<cfif y IS 1>
<cfset selParaCombinations = selQ>
</cfif>
</cfloop><!--- loop through paramters --->
<!--- build cartesian product of queries (this will show us all the needed combinations --->
<cfset qCounter = 0>
<cfloop list="#strQList#" index="qName">
<cfset qCounter ++>
<cfif qCounter GT 1>
<!--- combine into cartesian set. We can only do two queries at a time --->
<cfset qToAdd = stcQueries[qName]>
<cfquery name="selParaCombinations" dbtype="query">
SELECT DISTINCT *
FROM selParaCombinations,qToAdd
</cfquery>
</cfif>
</cfloop>
<!--- determine max loop (bsoylu 03-30-2012) --->
<cfset intMaxLoop = selParaCombinations.RecordCount>
<cfif intMaxCases GT 0>
<cfset intMaxLoop = intMaxCases>
</cfif>
<cfoutput>
found #selParaCombinations.RecordCount# test combinations
<cfif intMaxCases GT 0 AND intMaxCases LT selParaCombinations.RecordCount>
<b>only #intMaxCases#</b> will be generated
</cfif>
<br>
</cfoutput>
<cfflush>
<!--- loop and generate test for file (bsoylu 03-29-2012) --->
<cfset intTestCount = 0>
<cfset intFileCount = 0>
<cfloop query="selParaCombinations" endrow="#intMaxLoop#">
<cfset tf="">
<cfset intTestCount ++ >
<!--- create para structure (bsoylu 03-29-2012) --->
<cfset stcPara = {}>
<cfloop list="#selParaCombinations.ColumnList#" index="idxCol">
<cfset strVal=Evaluate("selParaCombinations.#idxCol#")>
<!--- check whether we need to further process the content (bsoylu 03-29-2012) --->
<cfif strVal neq "NULL">
<cftry>
<cfif strVal is "">
<cfset stcPara[idxCol] = strVal>
<cfelseif Left(strVal,1) IS ":">
<cfset stcPara[idxCol] = DeserializeJSON( Right(strVal,Len(strVal)-1))>
<cfelseif Left(strVal,1) IS "=">
<cfset stcPara[idxCol] = Evaluate( Right(strVal,Len(strVal)-1))>
<cfelse>
<cfset stcPara[idxCol] = strVal>
</cfif>
<cfcatch>
<hr>
<cfoutput>could not interpret #Right(strVal,Len(strVal)-1)#</cfoutput>
</hr>
<cfabort>
</cfcatch>
</cftry>
</cfif>
</cfloop>
<!--- create function for this para combination (bsoylu 03-29-2012) --->
<cfset strJSON = SerializeJSON(stcPara)>
<cfset tf=tf & crlf>
<cfset tf=tf & crlf & tab & '<cffunction name="test_#strMethod#_#NumberFormat(selParaCombinations.CurrentRow,"00000")#" >''>
<cfset tf=tf & crlf & tab & tab & '<cfset var strJSONPara =''' & strJSON &'''>''>
<cfset tf=tf & crlf & tab & tab & '<cfset var response ="">''>
<cfset tf=tf & crlf & tab & tab & '<cfset var argCol = DeserializeJSON(strJSONPara)>''>
<cfset tf=tf & crlf & tab & tab & '<cfinvoke argumentcollection="##argCol##" component="#strName#" method="#strMethod#" returnvariable="response"/>''>
<cfset tf=tf & crlf & tab & tab & '<cfset assertNotEquals( "",response,SerializeJSON(response) & " - failed call with' & Replace(strJSON,'"','""','ALL') & '")>''>
<cfset tf=tf & crlf & tab & '</cffunction>''>
<cfset tf=tf & crlf>
<!--- add to test file content (bsoylu 03-29-2012) --->
<cfset tc=tc & crlf & tf>
<cfif intTestCount mod intMaxCasesPerFile IS 0>
<!--- write what we have so far (bsoylu 04-03-2012) --->
<cfset intFileCount++>
<cfset strModTestCaseName = strTestCaseName & "_File" & intFileCount>
<cfset tC = tC & crlf & '</cfcomponent>''>
<!--- set numbered File names (bsoylu 04-03-2012) --->
<cfset strModTestCaseFileName= strTestCaseDirPath & "\#strModTestCaseName#.cfc">
<cfset ArrayAppend(arrTestCaseNames,strModTestCaseName)>
<cffile action="WRITE" file="#strModTestCaseFileName#" output="#tc#" addnewline="No">
<cfoutput>generated sub case file: #strModTestCaseName#<br></cfoutput>
<!--- reset tc for next file (bsoylu 04-03-2012) --->
<cfset genStartTc("_File" & intFileCount + 1)>
</cfif>
</cfloop> <!--- set parameters (bsoylu 03-29-2012) --->
</cfif> <!--- we have parameters --->
<!--- complete the last file (bsoylu 04-03-2012) --->
<cfset tC = tC & crlf & '</cfcomponent>''>
<cfset ArrayAppend(arrTestCaseNames,strTestCaseName)>
<!--- write test case --->
<cffile action="WRITE" file="#strTestCaseFileName#" output="#tc#" addnewline="No">
<cfoutput>generated last case file: #strTestCaseName#<br></cfoutput>
</cfif> <!--- public function --->
</cfloop> <!--- loop through functions --->
<!--- generate test suite file --->
<cfset genTestSuite()>
<cfset blnComplete = true>
</cfif>
<hr>
<cfcatch type="Any">
<font color="red" size="+2">
There was an error. Please ensure that your component is located in the correct directory: <BR>
</font>
<table cellspacing="2" cellpadding="2" border="1">
<tr>
<td><cfdump var="#cfcatch#">
</td>
</tr>
</table>
<HR color="#ff0000" noshade>
</cfcatch>
</cftry>
</CFIF>
<cffunction name="genTestSuite">
<cfset var tcName ="">
<cfset tf = crlf>
<cfset tf=tf & crlf & '<cfparam name="URL.output" default="extjs">''>
<cfset tf=tf & crlf & '<cfset testSuitePath = "mxunit.framework.TestSuite" >''>
<cfset tf=tf & crlf & '<cfset testSuite = createObject("component", testSuitePath).TestSuite() >''>
<cfloop from="1" to="#ArrayLen(arrTestCaseNames)#" index="idxArr">
<cfset tcName = arrTestCaseNames[idxArr]>
<cfset tf=tf & crlf & '<cfset uTest = createObject("component", "#strTestCaseDirName#.#tcName#")>''>
<cfset tf=tf & crlf & '<cfset testSuite.addAll("#tcName#", uTest) >''>
<cfset tf=tf & crlf>
</cfloop>
<cfset tf=tf & crlf>
<cfset tf=tf & crlf & '<cfset results = testSuite.run() >''>
<cfset tf=tf & crlf & '<cfset out = results.getResultsOutput(URL.output)>''>
<cfset tf=tf & crlf & '<cfif NOT IsSimpleValue(out)>''>
<cfset tf=tf & crlf & tab & '<cfdump var="##out##">''>
<cfset tf=tf & crlf & '<cfelse>''>
<cfset tf=tf & crlf & tab & '<cfoutput>'##out##</cfoutput>'>
<cfset tf=tf & crlf & '</cfif>''>
<!--- loop through and add tests (bsoylu 03-29-2012) --->
<cffile action="WRITE" output="#tf#" file="#strTestSuitePath#" addnewline="No">
</cffunction>
<cffunction name="genStartTc">
<cfargument name="strModifier" type="string" default="" hint="modifier for component name">
<!--- start testCase tC --->
<cfset tC = '<cfcomponent displayname="MxunitTestCase_#strTestCaseName##Arguments.strModifier#" extends="mxunit.framework.TestCase">''>
<!--- empty call no parameters --->
<cfif Arguments.strModifier IS "">
<cfset tc=tc & crlf & tab & '<cffunction name="testNoParams_#strMethod#" >''>
<cfset tc=tc & crlf & tab & tab & '<cfset var response ="">''>
<cfset tc=tc & crlf & tab & tab & '<cfinvoke component="#strName#" method="#strMethod#" returnvariable="response"/>''>
<cfset tc=tc & crlf & tab & tab & '<cfset assertNotEquals( "",response,"No parameter call failed")>''>
<cfset tc=tc & crlf & tab & '</cffunction>''>
</cfif>
</cffunction>
<cfif IsDefined("blnComplete") and blnComplete>
<font size="+2" color="#008000">
<cfoutput>
Successfully generated Test Suite for Component [#UCase(Form.objectName)#]. <BR>
Test suite file : <a href="#strDirPrefix#/#strTestSuiteName#" target="_blank" title="run tests">[#strTestSuitePath#]</a> <BR>
Please review and adjust specific test cases.
</cfoutput>
</font>
<cfabort>
</cfif>
<h2>Welcome</h2>
<BR>
This program will help you generate mxUnit test cases.
It requires access to components to be analyzed.
<BR>
After initial generation you should make changes as the assertion are generic.
The objective is provide broad test coverage.
<HR>
Behavior Notes <BR>
<PRE>
All files will be generated in the "unitTests" subdirectory based on the initial directory this template is run from
Will attempt to generate all permutations possible.
Random values
Break Values
Test File Name:
TestCases_[componentName]\Test_[componentName]_[functionName].cfc
If multiple files are generated per function, a File[n] postfix wil be added except the last one:
TestCases_[componentName]\Test_[componentName]_[functionName]_File[n].cfc
An overall TestSuite will be generated
Test Suite name:
TestSuite_[componentName].cfm
</PRE>
<form method="post" action="GenerateTestsPublic.cfm">
<table cellspacing="2" cellpadding="2" border="0">
<tr>
<td>Component Path from the webroot (e.g. component located [root]/myweb/cfcs/foo.cfc would be myweb.cfcs.foo)</td>
<td><input type="Text" name="objectName" value=""></td>
</tr>
<tr>
<td>Max Number of Test Cases per File (same function)</td>
<td><input type="Text" name="maxTestsPerFile" value="1000"> (multiple files will be generated if more than this number)</td>
</tr>
<tr>
<td>Max Number of Test Cases for a Function</td>
<td><input type="Text" name="maxTests" value="5000"></td>
</tr>
<tr>
<td></td>
<td><input type="submit" name="submit" value="submit"></td>
</tr>
</table>
</form>
</body>
</html>
Cheers,
B.
3 comments:
I just tried your script. First I have to fix synthax errors (quotes). After that it worked. I think this is great tool for the start but it need some work to be smarter.
Nice job after all.
Greetings.
Glad to hear it was helpful.
Best,
B.
Hi, will you continue developing this tool? It could be a very interesting modul aka "Automatic Test Generator".
Post a Comment