[X3D-Public] new X3D quality assurance tests using regular expressions (regexes)

[Don Brutzman]

Difficult flaws to detect in .x3d scenes include floating point
arrays that are garbled by missing spaces, or numbers that have
extraneous leading zeroes.  These can cause bad rendering problems.
Browsers are inconsistent about detecting such errors, and often
do not provide diagnostics that such problems exist.

Regular expressions provide a way to find string combinations that
meet a certain pattern.  This can provide a powerful diagnostic tool.

Terry Norbraten and I have worked up some regular expressions to detect
these two problems.  The tests are implemented in the build.xml files
that are used to prepare the many X3D scenes in our example archives.

Suggestions for other difficult-to-detect scene pathologies are welcome.
Further detail on regular expressions can be found at
	http://en.wikipedia.org/wiki/Regular_expression
	http://www.regular-expressions.info
	http://regexlib.com

The X3D working group is dedicated alternating weekly teleconferences
to discuss X3D authoring issues.  We hope to identify best practices,
help fix problems, and give browser companies a prioritized list of
common browser bugs and inconsistencies.  Fixing errors in content
is a big step towards ensuring that the bug isn't in the scene itself.

The following code block is written for Ant, and includes the two
regexes being used so far.  The regexes themselves are quite terse,
quite fast to perform, and supported in all modern programming
languages.  So X3D tools may want to implement them.  The expressions
can be cryptic but are easy enough to follow once you get the hang of it.

float with two decimal points:
	(\s|"|')(\+|-)?\d*\.\d+(E|e)?(\+|-)?\.\d*(\s|"|')

number with excess leading zeroes:
(\s|"|')(\+|-)?0\d+(\.\d*)?(E|e)?(\+|-)?(\.\d*)?(\s|"|')
    
    <target name="processScenes.regularExpressionChecks" description="find offensive lines using regexes">
        <concat>
            <!-- destfile not specified, so concat goes to console -->
            <header filtering="no" trimleading="yes">Lines that contain malformed floats:
                ==============================================
            </header>
            <path id='x3dFiles' description="contained x3d files">
                <fileset dir=".">
                    <include name="*/*.x3d"/>
                    <include name="*/*/*.x3d"/>
                    <exclude name="**/_archive/*.x3d"/>
                    <exclude name="**/hide/*.x3d"/>
                    <exclude name="**/originals/*.x3d"/>
                    <exclude name="**/.svn/*.x3d"/>
                </fileset>
            </path>
            <filterchain>
                <linecontainsregexp>
                    <regexp pattern="(?<!address=)(?<!Section)(?<!CosmoPlayer)(\s|"|')(\+|-)?\d*\.\d+(E|e)?(\+|-)?\.\d*(\s|"|')"/>
                    <!-- (?<!address=) is a "negative lookbehind" to filter out strings starting with address= -->
                    <!-- http://stackoverflow.com/questions/977251/regular-expressions-and-negating-a-whole-character-group -->
                </linecontainsregexp>
            </filterchain>
            <footer filtering="no" trimleading="yes">==============================================
            </footer>
        </concat>
        <concat>
            <!-- destfile not specified, so concat goes to console -->
            <header filtering="no" trimleading="yes">Lines that contain numbers with leading zeros:
                ==============================================
            </header>   
            <path refid='x3dFiles'/>
            <filterchain>
                <linecontainsregexp>
                    <regexp pattern="(\s|"|')(\+|-)?0\d+(\.\d*)?(E|e)?(\+|-)?(\.\d*)?(\s|"|')"/>
                    <!-- (?<!latitude)(?<!longitude) -->
                </linecontainsregexp>
            </filterchain>
            <footer filtering="no" trimleading="yes">==============================================
            </footer>   
        </concat>
    </target>

I've added a test scene to ensure these are working at
http://www.web3d.org/x3d/content/examples/Basic/development/TestRegularExpressionChecks.x3d

Example output to check 690 scenes in X3D Basic Archive follows, which
shows that prior errors are now cleaned up and the test cases are
properly caught.

processScenes.regularExpressionChecks:
Lines that contain malformed floats:
==============================================
    <ScalarInterpolator DEF='MergedFloatsTest' key='0 0.250.5 0.75 1' keyValue='0 1 2 3 4 5'/>
==============================================
Lines that contain numbers with leading zeros:
==============================================
    <ScalarInterpolator DEF='LeadingZerosTest' key='0 0.25 0.5 0.75 1' keyValue='0 01 2 3 4 5'/>
==============================================
BUILD SUCCESSFUL (total time: 10 seconds)

TODO:
- add this capability to X3D-Edit Quality Assurance (QA) checks
- build results will appear online tonite via
	https://savage.nps.edu/jenkins/
	X3dExamplesBasic
	X3dExamplesConformanceNist
	X3dExamplesSavage
	X3dExamplesVrml2.0Sourcebook
	X3dExamplesX3dForAdvancedModeling
	X3dExamplesX3dForWebAuthors
- build a server-side page that X3D authors can use to perform
	regex checks and other validations
- detect and fix X3D scenes with problems!

all the best, Don
-- 
Don Brutzman  Naval Postgraduate School, Code USW/Br       brutzman at nps.edu
Watkins 270,  MOVES Institute, Monterey CA 93943-5000 USA   +1.831.656.2149
X3D graphics, virtual worlds, navy robotics http://faculty.nps.edu/brutzman