FCODE suite: Difference between revisions

From OpenBIOS
Jump to navigation Jump to search
No edit summary
 
(58 intermediate revisions by 3 users not shown)
Line 1: Line 1:
== What is the OpenBIOS FCODE Suite? ==
== What is the OpenBIOS FCODE Suite? ==


OpenBIOS provides a sophisticated set of FCODE utilities: A tokenizer, a detokenizer, and a PCI rom header utility.
OpenBIOS provides a sophisticated set of FCODE utilities:  
* the tokenizer '''toke'''
* the detokenizer '''detok'''
* and a PCI rom header utility.
* a portable implementation of forth local values


In August 2006 David Paktor, of the IBM Corporation, sent us a comprehensive set of contributions to the OpenBIOS FCODE suite. We have not yet fully proofread and integrated these changes into the official OpenBIOS development tree, but in the meantime we want to offer you the opportunity to download the files directly.
These files are offered without any warranty. If you experience problems, please contact the [[Mailinglist|OpenBIOS mailinglist]].


These files are offered without any warranty. If you experience problems, please contact the OpenBIOS mailinglist.


== Downloading the OpenBIOS FCode Suite ==
== Downloading the OpenBIOS FCode Suite ==
'''The latest version of the OpenBIOS FCode Suite is 1.0.2 (released 2006-10-30)'''


=== Source Code ===
=== Source Code ===


Modified [http://www.openbios.org/fcodesuite/Tokenizer Sources and Makefiles]. Available as a Tar-File:  [http://www.openbios.org/fcodesuite/TokzrNewSrcFiles.tar.bz2 TokzrNewSrcFiles.tar.bz2]
View the [https://github.com/openbios/fcode-utils Sources] online. Available as a ZIP-File:  [https://github.com/openbios/fcode-utils/archive/v1.0.2.zip FCode-utils 1.0.2]
 
See the [https://raw.githubusercontent.com/openbios/fcode-utils/v1.0.2/ChangeLog ChangeLog] for a list of changes since 1.0.1.
A hand crafted html document describes [http://www.openbios.org/data/visiblediffs/V01_versus_V2x/ some more changes].


=== Documentation ===
=== Documentation ===
Line 19: Line 27:


The documents are User's Guides to:
The documents are User's Guides to:
* the [http://www.openbios.org/fcodesuite/Documents/TokenizerNewFeatures.htm New Features of the Tokenizer],
* the [http://www.openbios.org/data/fcodesuite/Documents/toke.html New Features of the Tokenizer],
* [http://www.openbios.org/fcodesuite/Documents/DeTokenizer.htm the Detokenizer], and
* [http://www.openbios.org/data/fcodesuite/Documents/detok.html the Detokenizer], and
* the [http://www.openbios.org/fcodesuite/Documents/TokenizerLocalValues.htm Local Values feature], which is mentioned briefly in the Tokenizer User's Guide and described fully in the Local Values document.
* the [http://www.openbios.org/data/fcodesuite/Documents/localvalues.html Local Values feature], which is mentioned briefly in the Tokenizer User's Guide and described fully in the Local Values document.
 
These documents are also part of the source code repository.


Available as a Tar-File:  [http://www.openbios.org/fcodesuite/Documents.tar.bz2 Documents.tar.bz2]
There is also doxygen generated documentation available
* [http://openbios.org/data/toke/toke-0.6.10 doxygen documentation for toke 0.6.10]
* [http://openbios.org/data/toke/toke-1.0 doxygen documentation for toke 1.0.0]
* TODO: doxygen documentation for toke 1.0.2


=== Executables for three platforms ===
=== Executables for three platforms ===


There are three programs: the Tokenizer, the Detokenizer and the ROMHeaders utility. ([http://www.openbios.org/fcodesuite/Binaries Binaries])
While you can find a couple of executables here we strongly recommend that you compile the FCode toolchain from the sources above so you gain from the integration work and fixes that have been done since these executables have been created.
 
There are three programs: the Tokenizer, the Detokenizer and the ROMHeaders utility. ([http://www.openbios.org/data/fcodesuite/Binaries Binaries])


There is a version for each of three platforms (i.e., combinations of Processor and O/S): Cygwin running on an X86, GNU Linux running on a Power-PC, and AIX running on a Power-PC.
There is a version for each of three platforms (i.e., combinations of Processor and O/S): Cygwin running on an X86, GNU Linux running on a Power-PC, and AIX running on a Power-PC.
Line 39: Line 54:
Under each are sub-directories for the different platforms, within which the executable images reside.
Under each are sub-directories for the different platforms, within which the executable images reside.


Available as a Tar-File:  [http://www.openbios.org/fcodesuite/Binaries.tar.bz2 Binaries.tar.bz2]
All binaries are also available in a single Tar-File:  [http://www.openbios.org/data/fcodesuite/Binaries.tar.bz2 Binaries.tar.bz2]


=== Local Value Support ===
=== Local Value Support ===


Includ-able tokenizer-source files for [http://www.openbios.org/fcodesuite/LocalValues Local Values Support] (explained in one of the User's Guide documents). Five files:  
Includ-able tokenizer-source files for [https://github.com/openbios/fcode-utils/tree/master/localvalues Local Values Support] (explained in one of the User's Guide documents). Five files:  
* One supplies the [http://www.openbios.org/fcodesuite/LocalValues/LocalValuesSupport.fth basic functionality]
* One supplies the [https://github.com/openbios/fcode-utils/tree/master/localvalues/LocalValuesSupport.fth basic functionality]
* the second adds a [http://www.openbios.org/fcodesuite/LocalValues/LocalValuesDevelSupport.fth development-time facility]
* the second adds a [https://github.com/openbios/fcode-utils/tree/master/localvalues/LocalValuesDevelSupport.fth development-time facility]
* the third generates a variant behavior ([http://www.openbios.org/fcodesuite/LocalValues/GlobalLocalValues.fth "Global" scope] rather than scope limited to a single Device-Node)  
* the third generates a variant behavior ([https://github.com/openbios/fcode-utils/tree/master/localvalues/GlobalLocalValues.fth "Global" scope] rather than scope limited to a single Device-Node)  
* and the fourth [http://www.openbios.org/fcodesuite/LocalValues/GlobalLocalValuesDevel.fth combines the "Global" variant behavior with the development-time facility].  
* and the fourth [https://github.com/openbios/fcode-utils/tree/master/localvalues/GlobalLocalValuesDevel.fth combines the "Global" variant behavior with the development-time facility].  
* The fifth [http://www.openbios.org/fcodesuite/LocalValues/TotalLocalValuesSupport.fth allows the choice of combinations to be governed by command-line switches], and is probably the best to use with Makefiles in commercial development and production environments.
* The fifth [https://github.com/openbios/fcode-utils/tree/master/localvalues/TotalLocalValuesSupport.fth allows the choice of combinations to be governed by command-line switches], and is probably the best to use with Makefiles in commercial development and production environments.


There is commentation in each one explaining how it is to be used.
There is commentation in each one explaining how it is to be used.


Available as a Tar-File:  [http://www.openbios.org/fcodesuite/LoclValSupport.tar.bz2 LoclValSupport.tar.bz2]
Available as part of the OpenBIOS FCODE suite.


=== Todo ===
=== Todo ===
Line 58: Line 73:
A list of "Still To Be Done" items, excerpted from the commentation in the Sources
A list of "Still To Be Done" items, excerpted from the commentation in the Sources


The source files have, scattered among their commentation, an occasional item discussing a feature or implementation detail that might be worth attention in future revisions. This file is a [http://www.openbios.org/fcodesuite/StillToBeDones_List collection of all of them] in a single convenient location.
The source files have, scattered among their commentation, an occasional item discussing a feature or implementation detail that might be worth attention in future revisions.  


Available as a Tar-File:  [http://www.openbios.org/fcodesuite/StillToBeDonesList.tar.bz2 StillToBeDonesList.tar.bz2]
This file is a [https://github.com/openbios/fcode-utils/tree/master/TODO collection of all of them] in a single convenient location.


== Unit-Test Suite ==
== Unit-Test Suite ==


=== The suite of unit-test cases ===
=== The suite of unit-test cases ===
This is the accumulation of test-cases that were created in the course of development. Some of these are a straightforward invocation of a feature, others are convoluted combinations of features whose interaction needed to be carefully watched, and still others are collections of coding errors, for purposes of verifying the Error-detection capabilities of the Tokenizer. They are grouped into sub-directories representing broad categories.
This is the [https://github.com/openbios/fcode-utils/tree/master/testsuite accumulation of test-cases] that were created in the course of development. Some of these are a straightforward invocation of a feature, others are convoluted combinations of features whose interaction needed to be carefully watched, and still others are collections of coding errors, for purposes of verifying the Error-detection capabilities of the Tokenizer. They are grouped into sub-directories representing broad categories.


One small set of items might be of particular interest: In a subdirectory called TokMisc/OldTkzr are two test-case source files, called ContErr0.fth and SimplStrEscs.fth together with output files that have .OldTkzr. in their names. These are the "Log" file, FCode binary output, and DeTokenization of the FCode binary, respectively, as they appear after having been run through the "old" tokenizer (version 0.6.10, dated 2005-10-15).
Run the unit-test cases with


The proper operation of the Test-suite has some dependency on the directory structure. Specifically, it expects the LocalValues directory, containing the Local Values Support files, to be a sibling of the parent (an "uncle" if you will) of the directory in which the Test-suite resides. There are sym-links that will clarify this.
$ make tests


Available as a Tar-File:  TestSrc.tar.bz2
=== Test Tools ===
The tools to run the Unit-Test Suite as a batch and examine the results for changes relative to the results from a previous run.


=== Test Tools ===
The process of manually running a unit-test and comparing against the previous output, after every change, became unwieldy, especially when it came to running the entire suite of tests. These scripts were developed to automate both processes:
The tools to run the Unit-Test Suite as a batch and examine the results for changes relative to the results from a previous run


The process of manually running a unit-test and comparing against the previous output, after every change, became unwieldy, especially when it came to running the entire suite of tests. These scripts were developed to automate both processes: AutoExec automates the execution, and AutoCompare automates the comparison. Again, there is commentation in each explaining how it is used.
* [https://github.com/openbios/fcode-utils/tree/master/testsuite/AutoExec AutoExec] automates the execution, and  
* [https://github.com/openbios/fcode-utils/tree/master/testsuite/AutoCompare AutoCompare] automates the comparison.


Available as a Tar-File:  TestTools.tar.bz2
There is commentation in each explaining how it is used.


== Unit-Test Suite Logs ==
== Unit-Test Suite Logs ==
Line 87: Line 103:
All in all, five or six file differences will be expected to be reported by AutoCompare.
All in all, five or six file differences will be expected to be reported by AutoCompare.


* The results from a run of the Unit-Test Suite on the [http://www.openbios.org/fcodesuite/TestLogs/X86_Cygwin X86/Cygwin] platform. Available as a Tar-File: [http://www.openbios.org/fcodesuite/X86_Cygwin_TestLog.tar.bz2 X86_Cygwin_TestLog.tar.bz2]
* The results from a run of the Unit-Test Suite on the [https://github.com/openbios/fcode-utils/tree/master/testlogs/testlogs-x86-cygwin X86/Cygwin] platform.
 
* The results from a run of the Unit-Test Suite on the [https://github.com/openbios/fcode-utils/tree/master/testlogs/testlogs-ppc-linux PowerPC/Linux] platform.
 
* The results from a run of the Unit-Test Suite on the [https://github.com/openbios/fcode-utils/tree/master/testlogs/testlogs-ppc-aix PowerPC/AIX] platform.
 
== Coverage Reports ==
 
The test suite has been run using gcov/lcov to produce graphical code coverage reports.
 
* [http://www.openbios.org/data/coverage-fcode-suite-1.0.2/ coverage report for FCODE suite 1.0.2]
* [http://www.openbios.org/data/coverage-toke-1.0.2/ coverage report for toke 1.0.2]
* [http://openbios.org/data/toke/coverage/ coverage report for toke 1.0.0]
 
 
== Kudos and Thanks ==


* The results from a run of the Unit-Test Suite on the [http://www.openbios.org/fcodesuite/TestLogs/PPC_GNU_Linux Power-PC/Linux] platform. Available as a Tar-File: [http://www.openbios.org/fcodesuite/PPC_GNU_Linux_TestLog.tar.bz2 PPC_GNU_Linux_TestLog.tar.bz2]
<div style="margin-top:0.5em; margin-bottom:0.5em; padding:0.5em 0.5em 0.5em 0.5em; background-color:#efefff; align:right; border:1px solid #aabbcc;">
The OpenBIOS FCODE Suite has been significantly enhanced to meet commercial grade requirements by [mailto:David@paktor.biz David Paktor]. Regarding to code readability and stability, he made the FCODE suite the best part of OpenBIOS. And it's probably the best FCODE development environment out there. '''Thank you, David!'''
</div>


* The results from a run of the Unit-Test Suite on the [http://www.openbios.org/fcodesuite/TestLogs/PPC_AIX Power-PC/AIX] platform. Available as a Tar-File: [http://www.openbios.org/fcodesuite/PPC_AIX_TestLog.tar.bz2 PPC_AIX_TestLog.tar.bz2]
   
__NOEDITSECTION__

Latest revision as of 01:35, 16 January 2016

What is the OpenBIOS FCODE Suite?

OpenBIOS provides a sophisticated set of FCODE utilities:

  • the tokenizer toke
  • the detokenizer detok
  • and a PCI rom header utility.
  • a portable implementation of forth local values

These files are offered without any warranty. If you experience problems, please contact the OpenBIOS mailinglist.


Downloading the OpenBIOS FCode Suite

The latest version of the OpenBIOS FCode Suite is 1.0.2 (released 2006-10-30)

Source Code

View the Sources online. Available as a ZIP-File: FCode-utils 1.0.2

See the ChangeLog for a list of changes since 1.0.1. A hand crafted html document describes some more changes.

Documentation

There are three documents, all in html format, plus a sub-directory of templates that provide common formatting support.

It is important that these be kept in the same directory, as there are some links from one file to another.

The documents are User's Guides to:

These documents are also part of the source code repository.

There is also doxygen generated documentation available

Executables for three platforms

While you can find a couple of executables here we strongly recommend that you compile the FCode toolchain from the sources above so you gain from the integration work and fixes that have been done since these executables have been created.

There are three programs: the Tokenizer, the Detokenizer and the ROMHeaders utility. (Binaries)

There is a version for each of three platforms (i.e., combinations of Processor and O/S): Cygwin running on an X86, GNU Linux running on a Power-PC, and AIX running on a Power-PC.

There are two variants of each version: One that has level-2 Optimization and one that has no optimization at all, which I provided for purposes of Debugging. Optimization causes some routines and variables to become obscured and inaccessible to debuggers, and also re-arranges the sequence of execution in a way that can become confusing during single-stepping.

And finally, for each, there is a "stripped" and an "unstripped" executable image. The "unstripped" image has an extension of "unstripped"; the "stripped" image has no extension.

There are separate directories for the Debug and Optim(ized Level)2 variants.

Under each are sub-directories for the different platforms, within which the executable images reside.

All binaries are also available in a single Tar-File: Binaries.tar.bz2

Local Value Support

Includ-able tokenizer-source files for Local Values Support (explained in one of the User's Guide documents). Five files:

There is commentation in each one explaining how it is to be used.

Available as part of the OpenBIOS FCODE suite.

Todo

A list of "Still To Be Done" items, excerpted from the commentation in the Sources

The source files have, scattered among their commentation, an occasional item discussing a feature or implementation detail that might be worth attention in future revisions.

This file is a collection of all of them in a single convenient location.

Unit-Test Suite

The suite of unit-test cases

This is the accumulation of test-cases that were created in the course of development. Some of these are a straightforward invocation of a feature, others are convoluted combinations of features whose interaction needed to be carefully watched, and still others are collections of coding errors, for purposes of verifying the Error-detection capabilities of the Tokenizer. They are grouped into sub-directories representing broad categories.

Run the unit-test cases with

$ make tests

Test Tools

The tools to run the Unit-Test Suite as a batch and examine the results for changes relative to the results from a previous run.

The process of manually running a unit-test and comparing against the previous output, after every change, became unwieldy, especially when it came to running the entire suite of tests. These scripts were developed to automate both processes:

There is commentation in each explaining how it is used.

Unit-Test Suite Logs

These can be used as base-lines for comparison against future versions, or, if so be, versions compiled for additional platforms.

Note that a comparison of these against each other will not yield exact identity. Some of the test-cases, for instance, code the current date and time, others display a complete file-path, and still one other attempts to load a file for encoding using a syntax that is erroneous on some O/S's but not on others.

All in all, five or six file differences will be expected to be reported by AutoCompare.

  • The results from a run of the Unit-Test Suite on the X86/Cygwin platform.
  • The results from a run of the Unit-Test Suite on the PowerPC/Linux platform.
  • The results from a run of the Unit-Test Suite on the PowerPC/AIX platform.

Coverage Reports

The test suite has been run using gcov/lcov to produce graphical code coverage reports.


Kudos and Thanks

The OpenBIOS FCODE Suite has been significantly enhanced to meet commercial grade requirements by David Paktor. Regarding to code readability and stability, he made the FCODE suite the best part of OpenBIOS. And it's probably the best FCODE development environment out there. Thank you, David!