= X12Parser - a library to manipulate X12 structures using native Ruby syntax

$Id: README 92 2009-05-13 22:12:10Z ikk $

*WARNING* <tt>The project is in development. Contributors are welcome.</tt>

Project home is at http://rubyforge.org/projects/x12parser/. Please
note, this is a different project from {Chris Parker's port}[http://rubyforge.org/projects/x12-parser/] of
{X12::Parser Perl module}[http://search.cpan.org/~prasad/X12-0.09/lib/X12/Parser.pm].

== The goal

The idea is to access X12 messages directly from Ruby, i.e., using a
syntax like

  message.L1000.L1010[1].AK4.DataElementReferenceNumber

This syntax can be used to get and set any field of an X12 message and
it makes X12 parsing much more straightforward and self-documenting.

== The problem

X12 is a set of "standards" possessing all the elegance of an elephant
designed by committee, and quite literally so, see http://www.x12.org.
X12 defines rough syntax for specifying text messages, but each of
more than 300 specifications defines its own message structure. While
messages themselves are easy to parse with a simple tokenizer, their
semantics is heavily dependent on the domain. For example, this is
X12/997 message conveying "Functional Acknowledgment":

  ST*997*2878~AK1*HS*293328532~AK2*270*307272179~AK3*NM1*8*L1010_0*8~
  AK4*0:0*66*1~AK4*0:1*66*1~AK4*0:2*66*1~AK3*NM1*8*L1010_1*8~AK4*1:0*
  66*1~AK4*1:1*66*1~AK3*NM1*8*L1010_2*8~AK4*2:0*66*1~AK5*R*5~AK9*R*1*
  1*0~SE*8*2878~

I.e., X12 defines an alphabet and somewhat of a dictionary - not a
grammar or semantics for each particular data interchange
conversation. Because of many entrenched implementations and
government mandates, the X12 is not going to die anytime soon,
unfortunately.

The message above can be easily represented in Ruby as a nested array:

 m = [
      ['ST', '997', '2878'],
      ['AK1', 'HS', '293328532'],
      ['AK2', '270', '307272179'],
      ['AK3', 'NM1', '8', 'L1010_0', '8'],
      ['AK4', '0:0', '66', '1'],
      ['AK4', '0:1', '66', '1'],
      ['AK4', '0:2', '66', '1'],
      ['AK3', 'NM1', '8', 'L1010_1', '8'],
      ['AK4', '1:0', '66', '1'],
      ['AK4', '1:1', '66', '1'],
      ['AK3', 'NM1', '8', 'L1010_2', '8'],
      ['AK4', '2:0', '66', '1'],
      ['AK5', 'R', '5'],
      ['AK9', 'R', '1', '1', '0'],
      ['SE', '8', '2878'],
     ]

but it will not help any since, say, segment 'AK4' is ambiguously
defined and its meaning not at all obvious until the message's
structure is interpreted and correct 'AK4' segment is found.

== The solution

=== Message structure

Each participant in EDI has to know the structure of the data coming
across the wire - X12 or no X12. The X12 structures are defined in
so-called Implementation Guides - thick books with all the data pieces
spelled out. There is no other choice, but to invent a
computer-readable definition language that will codify these
books. For familiarity sake we'll use XML. For example, the X12/997
message can be defined as

  <Definition>
    <Loop name="997">
      <Segment name="ST" min="1" max="1"/>
      <Segment name="AK1" min="1" max="1"/>
      <Loop name="L1000" max="999999" required="y">
        <Segment name="AK2" max="1" required="n"/>
        <Loop name="L1010" max="999999" required="n">
          <Segment name="AK3" max="1" required="n"/>
          <Segment name="AK4" max="99" required="n"/>
        </Loop>
        <Segment name="AK5" max="1" required="y"/>
      </Loop>
      <Segment name="AK9" max="1" required="y"/>
      <Segment name="SE"  max="1" required="y"/>
    </Loop>
  </Definition>

Namely, the 997 is a 'loop' containing segments ST (only one), AK1
(also only one), another loop L1000 (zero or many repeats), segments
AK9 and SE. The loop L1000 can contain a segment AK2 (optional) and
another loop L1010 (zero or many), and so on.

The segments' structure can be further defined as, for example,

  <Segment name="AK2">
    <Field name="TransactionSetIdentifierCode" required="y" min="3" max="3" validation="T143"/>
    <Field name="TransactionSetControlNumber"  required="y" min="4" max="9"/>
  </Segment>

which defines a segment AK2 as having two fields:
TransactionSetIdentifierCode and TransactionSetControlNumber. The
field TransactionSetIdentifierCode is defined as having a type of
string (default), being required, having length of minimum 3 and
maximum 3 characters, and being validated against a table T143. The
validation table is defined as

  <Table name="T143">
    <Entry name="100" value="Insurance Plan Description"/>
    <Entry name="101" value="Name and Address Lists"/>
    ...
    <Entry name="997" value="Functional Acknowledgment"/>
    <Entry name="998" value="Set Cancellation"/>
  </Table>

with entries having just names and values.

This message is fully flashed out in an example 'misc/997.xml' file,
copied from the ASC X12N 276/277 (004010X093) "Health Care
Claim Status Request and Response" National Electronic Data
Interchange Transaction Set Implementation Guide.

Now expressions like 

  message.L1000.L1010[1].AK4.DataElementReferenceNumber

start making sense of sorts, overall X12's idiocy notwithstanding - it's
a field called 'DataElementReferenceNumber' of a first of possibly
many segments 'AK4' found in the second repeat of the loop 'L1010'
inside the enclosing loop 'L1000'. The meaning of the value '66' found
in this field is still in the eye of the beholder, but, at least its
location is clearly identified in the message.


=== X12 Structure Definition Language

The syntax of the X12 structure definition language should be apparent
from the '997.xml' file enclosed with the package. A more detailed
description follows in Appendix A.

=== Parsing

Here is how to parse an X12/997 message (the source is in
example/parse.rb):

  require 'x12'

  # Read message definition and create an actual parser
  # by compiling the XML file
  parser = X12::Parser.new('misc/997.xml')

  # Define a test message to parse
  m997='ST*997*2878~AK1*HS*293328532~AK2*270*307272179~'\
  'AK3*NM1*8*L1010_0*8~AK4*0:0*66*1~AK4*0:1*66*1~AK4*0:2*'\
  '66*1~AK3*NM1*8*L1010_1*8~AK4*1:0*66*1~AK4*1:1*66*1~AK3*'\
  'NM1*8*L1010_2*8~AK4*2:0*66*1~AK5*R*5~AK9*R*1*1*0~SE*8*2878~'

  # Parse the message
  r = parser.parse('997', m997)

  # Access components of the message as desired

  # Whole ST segment: -> ST*997*2878~
  puts r.ST

  # One filed, Group Control Number of AK1 -> 293328532
  puts r.AK1.GroupControlNumber

  # Individual loop, namely, third L1010 sub-loop of
  # L1000 loop: -> AK3*NM1*8*L1010_2*8~AK4*2:0*66*1~
  puts r.L1000.L1010[2]

  # First encounter of Data Element Reference Number of the 
  # first L1010 sub-loop of L1000 loop -> 66
  puts r.L1000.L1010.AK4.DataElementReferenceNumber

  # Number of L1010 sub-loops in L1000 loop -> 3
  puts r.L1000.L1010.size

=== Generating

Here is how to perform a reverse operation and generate a well-formed
997 message (the source is in example/factory.rb):

  require 'x12'

  # Read message definition and create an actual parser
  # by compiling .d12 file
  parser = X12::Parser.new('misc/997.xml')

  # Make a new 997 message 
  r = parser.factory('997')

  #
  # Set various fields as desired
  #

  # Set fields directly
  r.ST.TransactionSetIdentifierCode = 997
  r.ST.TransactionSetControlNumber  = '2878'

  # Set fields inside a segment (AK1 in this case)
  r.AK1 { |ak1|
    ak1.FunctionalIdentifierCode = 'HS'
    ak1.GroupControlNumber       = 293328532
  }

  # Set fields deeply inside a segment inside 
  # nested loops (L1000/L1010/AK4 in this case)
  r.L1000.L1010.AK4.DataElementSyntaxErrorCode = 55
  r.L1000.AK2.TransactionSetIdentifierCode     = 270

  # Set nested loops
  r.L1000.L1010 {|l|
    l.AK3 {|s|
      s.SegmentIdCode      = 'NM1'
      s.LoopIdentifierCode = 'L1000D'
    }
    l.AK4 {|s|
      s.CopyOfBadDataElement = 'Bad element'
    }
  }

  # Add loop repeats
  r.L1000.repeat {|l1000|
    (0..1).each {|loop_repeat| # Two repeats of the loop L1010
      l1000.L1010.repeat {|l1010|
        l1010.AK3 {|s|
          s.SegmentIdCode                   = 'DMG'
          s.SegmentPositionInTransactionSet = 0
          s.LoopIdentifierCode              = 'L1010'
          s.SegmentSyntaxErrorCode          = 22
        } if loop_repeat == 0 # AK3 only in the first repeat of L1010
        (0..1).each {|ak4_repeat| # Two repeats of the segment AK4
          l1010.AK4.repeat {|s|
            s.PositionInSegment          = loop_repeat
            s.DataElementSyntaxErrorCode = ak4_repeat
          } # s
        } # ak4_repeat
      } # l1010
    } # loop_repeat

    l1000.AK5{|a|
      a.TransactionSetAcknowledgmentCode = 666
      a.TransactionSetSyntaxErrorCode4   = 999
    } # a
  } # l1000

  # Print the message as a string -> ST*997*2878~AK1*HS*293328532~
  # AK2*270*~AK3*NM1**L1000D~AK4***55*Bad element~AK5*~AK3*DMG*0*
  # L1010*22~AK4*0**0~AK4*0**1~AK4*1**0~AK4*1**1~AK5*666****999~
  # AK9****~SE**~
  puts r.render

== Download

The latest X12 library version can be downloaded from
http://rubyforge.org/frs/?group_id=7297

== Installation

You can install X12 library with the following command.

  % gem install X12

== License

X12 library is released under the Lesser GPL license, see
http://www.gnu.org/licenses/lgpl.txt

== Major deficiencies

* Validation is not implemented.
* Field types and sizes are ignored.
* No access methods for composites' fields.

== Support

Please use the following:

* forums on Rubyforge for general discussions, http://rubyforge.org/forum/?group_id=7297
* trackers to submit bugs or feature requests, http://rubyforge.org/tracker/?group_id=7297
* to contact the author, send mail to prelude rubyforge org

== Acknowledgments

The authors of the project were inspired by the following works:

1. The Perl X12 parser by Prasad Poruporuthan, http://search.cpan.org/~prasad/X12-0.09/lib/X12/Parser.pm
2. The Ruby port of the above by Chris Parker, http://rubyforge.org/projects/x12-parser/

== Appendix A. Structure definition language

The structure definition language uses XML to describe X12 message
format. A message definition can be in a single file or in several. If
the definition parser encounters an element (segment, composite, or
table), which has not been previously defined, it tries to load the
definition from the file with the same name and in the same
directory. For example, if a loop mentions a segment named 'ST' and
this segment is not defined, the parser will try to load and parse a
file called 'ST.xml'. This convention works for any name unless it
conflicts with a Microsoft's device name, see Appendix B.

Each element in a structure definition (except 'Definition') must have
an attribute called 'name'. It is used to set/get respective content
from Ruby. If an element's 'name' attribute cannot be a valid Ruby
identifier (for example, '270'), the parser will prepend the name with
'_' (underscore). I.e., this expression is not valid:

  @r.FG[1].270[1].ST.TransactionSetIdentifierCode

but his one is

  @r.FG[1]._270[1].ST.TransactionSetIdentifierCode

Each XML file has to have a single root element, one of the following:

=== Definition

The 'Definition' element can have nested loops, segments, composites,
and tables. It is used to provide 'artificial' root element for XML
document when several definitions are in one physical file. For
example, this is `misc/997single.xml' (edited for size):

  <Definition>
    <Segment name="ST">
      <Field name="TransactionSetIdentifierCode" min="3" max="3" validation="T143"/>
      <Field name="TransactionSetControlNumber"  min="4" max="9"/>
      <Field name="ImplementationConventionReference" required="y" min="1" max="35"/>
    </Segment>

    <Composite name="C030">
      <Field name="ElementPositionInSegment" type="long" required="n" min="1" max="2"/>
      <Field name="ComponentDataElementPositionInComposite" type="long" required="y" min="1" max="2"/>
      <Field name="RepeatingDataElementPosition" type="long" required="y" min="1" max="4"/>
    </Composite>

    <Segment name="AK1">
      <Field name="FunctionalIdentifierCode" min="2" max="2" validation="T479"/>
      <Field name="GroupControlNumber" type="long" min="1" max="9"/>
    </Segment>

    <Table name="T723">
      <Entry name="1" value="Mandatory data element missing"/>
      <Entry name="2" value="Conditional required data element missing."/>
      <!-- ... other entries -->
      <Entry name="13" value="Too Many Components"/>
    </Table>

    <!-- ... other segments or composites or tables -->

    <Loop name="997">
      <Segment name="ST" min="1" max="1"/>
      <Segment name="AK1" min="1" max="1"/>
      <Loop name="L1000" max="999999" required="y">
        <Segment name="AK2" max="1" required="n"/>
        <Loop name="L1010" max="999999" required="n">
          <Segment name="AK3" max="1" required="n"/>
          <Segment name="AK4" max="99" required="n"/>
        </Loop>
        <Segment name="AK5" max="1" required="y"/>
      </Loop>
      <Segment name="AK9" max="1" required="y"/>
      <Segment name="SE"  max="1" required="y"/>
    </Loop>

  </Definition>

This element does not have any attributes and cannot be addressed from Ruby's API.

=== Loop

The 'Loop' element is a main element to define either loops or whole
messages. Loops can have nested segments and other loops.

Note, that a segment defined inside a loop takes precedence over
previously defined segments. This is convenient if a special version
of a segment is required. For example, this is a general definition of
an 'ST' segment stored in a 'ST.xml' file:

  <Segment name="ST">
    <Field name="TransactionSetIdentifierCode" min="3" max="3" validation="T143"/>
    <Field name="TransactionSetControlNumber" min="4" max="9"/>
    <Field name="ImplementationConventionReference" required="y" min="1" max="35"/>
  </Segment>

If you want the X12 parser to look for a particular message type, say '997', do this:

  <Loop name="997">
    <Segment name="ST"  max="1">
      <Field name="TransactionSetIdentifierCode" const="997"/>
      <Field name="TransactionSetControlNumber" min="4" max="9"/>
    </Segment>
    <Segment name="AK1" max="1"/>
    <!-- ... the rest of the 997 definition -->
  </Loop>

A loop can have the following attributes:
* min - minimal number of repeats allowed, default is 0.
* max - maximum number of repeats allowed, default is 'inf' (infinite).
* required - if the loop is required (boolean), default is false. The true value implies min="1".
* comment - ignored

=== Segment

Segments can only have nested fields. Attributes are as follows:
* min - minimal number of repeats allowed, default is 0. Value min>0 implies required="y".
* max - maximum number of repeats allowed, default is 'inf' (infinite).
* required - if the segment is required (boolean), default is false. The true value implies min="1".
* comment - ignored

All attributed except 'name' are ignored in standalone definitions outside any loop.

=== Composite
Same as a segment.

=== Table

Tables can only have entries defined as name-value pairs. Order is
not important. Only required attribute is 'name'.

=== Field

A field cannot have any nested elements, but its attributes are very important:
* min - minimal number of characters allowed, default is 0. Value min>0 DOES NOT imply required="y" - the field can be missing, but may require a minimum length if present.
* max - maximum number of characters allowed, default is 'inf' (infinite).
* required - if the field is required (boolean), default is false. The true value DOES NOT imply min="1".
* type - one of the 'string' (default), 'integer', 'long', or 'double'. These abbreviations are also valid: 'str', 'int'.
* const - forces the field to have this value, if present.
* validation - the name of a validation table, if any.
* comment - ignored

== Appendix B. Microsoft's device file names

Apparently, in Microsoft's operating systems one cannot create a file
named like 'device_name.whatever', for example, 'CON.xml' is highly
illegal. For such cases, the X12 parser creates an exception and
expects 'CON_.xml' instead.

Here is the full device list according to Microsoft (see
http://support.microsoft.com/kb/74496):

   Name    Function
   ----    --------
   CON     Keyboard and display
   PRN     System list device, usually a parallel port
   AUX     Auxiliary device, usually a serial port
   CLOCK$  System real-time clock
   NUL     Bit-bucket device
   A:-Z:   Drive letters
   COM1    First serial communications port
   LPT1    First parallel printer port
   LPT2    Second parallel printer port
   LPT3    Third parallel printer port
   COM2    Second serial communications port
   COM3    Third serial communications port
   COM4    Fourth serial communications port