Code generation for multi-core patterns

[Stachelbeck]

When selecting to generate C code ocarina returns the following error:

/usr/local/ocarina/bin/ocarina -aadlv2 -g polyorb_hi_c -r AvionicsBusMgr.XPS8500 /home/stachelbeck/workspace/aadlwork/BusManagers/Plugin_Resources/deployment.aadl /home/stachelbeck/workspace/aadlwork/BusManagers/Plugin_Resources/bus_properties.aadl /home/stachelbeck/workspace/aadlwork/BusManagers/eNetManagers/MissionBus2.aadl /home/stachelbeck/workspace/aadlwork/BusManagers/Plugin_Resources/processor_properties.aadl /home/stachelbeck/workspace/aadlwork/BusManagers/eNetManagers/EWBus.aadl /home/stachelbeck/workspace/aadlwork/BusManagers/eNetManagers/AvionicsBusManagers.aadl /home/stachelbeck/workspace/aadlwork/BusManagers/eNetManagers/MissionBus1.aadl /home/stachelbeck/workspace/aadlwork/BusManagers/eNetManagers/Messages.aadl /home/stachelbeck/workspace/aadlwork/BusManagers/eNetManagers/MissionBus0.aadl
+========================== OCARINA BUG DETECTED =========================+
| Detected exception: SYSTEM.ASSERTIONS.ASSERT_FAILURE |
| Error: ocarina-backends-properties.adb:2452 |
| Please refer to the User's Guide for more details. |
+=========================================================================+

Exception name: SYSTEM.ASSERTIONS.ASSERT_FAILURE
Message: ocarina-backends-properties.adb:2452

What would be causing ocarina to abort with this kind of error?

Here is my main aadl file (AvionicsBusManagers.aadl):

package AvionicsBusManagers
public
with Deployment;
with Processor_Properties;
with Bus_Properties;

with EWBus;
with MissionBus0;
with MissionBus1;
with MissionBus2;

-- Hardware

virtual processor logical
end logical;

virtual processor implementation logical.core
properties
Deployment::Execution_Platform => LINUX64;
Priority_Range => 1 .. 255;
Scheduling_Protocol => (RMS);
end logical.core;

bus Ethernet
properties
Deployment::Transport_API => BSD_Sockets;
Bus_Properties::Available_Bandwidth => (10 MBytesps, 100 MBytesps, 1 GBytesps);
end Ethernet;

bus implementation Ethernet.XPS8500
end Ethernet.XPS8500;

processor Intel
features
ETH : requires bus access Ethernet;
end Intel;

processor implementation Intel.i7_3770
subcomponents
Cpu0 : virtual processor logical.core {Processor_Properties::Core_Id => 0;};
Cpu1 : virtual processor logical.core {Processor_Properties::Core_Id => 1;};
Cpu2 : virtual processor logical.core {Processor_Properties::Core_Id => 2;};
Cpu3 : virtual processor logical.core {Processor_Properties::Core_Id => 3;};
properties
Processor_Properties::Processor_Family => x86;
Processor_Properties::Endianess => Little_Endian;
Processor_Properties::Word_Length => 64 bits;
Processor_Properties::Processor_Frequency => 3400 MHz;
Deployment::Supported_Runtime => PolyORB_HI_C;
Deployment::Location => "127.0.0.1";
end Intel.i7_3770;

system AvionicsBusMgr
end AvionicsBusMgr;

system implementation AvionicsBusMgr.XPS8500
subcomponents
CPU : processor Intel.i7_3770;
EWBC : process EWBus::BusControllerMgr.impl {Deployment::Port_Number => 4000;};
M0RT : process MissionBus0::M0RemoteTerminalMgr.impl {Deployment::Port_Number => 4001;};
M1RT : process MissionBus1::M1RemoteTerminalMgr.impl {Deployment::Port_Number => 4002;};
M2RT : process MissionBus2::M2RemoteTerminalMgr.impl {Deployment::Port_Number => 4003;};
L0 : bus Ethernet.XPS8500;
connections
SC0 : bus access L0 <-> CPU.ETH;
SC1 : port M0RT.RTtoBCMsg -> EWBC.M0RTtoBCMsg {Actual_Connection_Binding => (reference(L0));};
properties
Actual_Processor_Binding => (reference(CPU.Cpu0)) applies to EWBC;
Actual_Processor_Binding => (reference(CPU.Cpu1)) applies to M0RT;
Actual_Processor_Binding => (reference(CPU.Cpu2)) applies to M1RT;
Actual_Processor_Binding => (reference(CPU.Cpu3)) applies to M2RT;
end AvionicsBusMgr.XPS8500;

end AvionicsBusManagers;

[yoogx]

Kenneth, PolyORB-HI/C generator expects a process to be bound to a processor, not a virtual processor. The assert failure is the following:

  pragma Assert (AINU.Is_Processor (P) or else AINU.Is_Device (P));

I investigate, but I guess we need to discuss in depth the patterns for code generation for multi-core systems. Your model implies you bind a process per core, I'd have thought of a model where you bind a process to a processor, and then allocate threads to core.

[Stachelbeck]

I noticed your update to ocarina and I download it and retested. These changes removed the assertion failure. This allowed me to proceed and uncover other problems not related to this one. These other problems I will submit later after I put together my findings. I did finally get socket connections between processes to work. But, I had to hand make changes to generated code (main.c's - comment out one line code), add a setsockopt (TCP_NODELAY) to polyorb-hi-c po_hi_driver_sockets.c, and correct the initialization of the struct in devicesconf.c (of the example you refered me to). So I had a successful weekend. Thanks for efforts.

[yoogx]

May you please detail why you had to patch the code, and the patch itself? This would help me provide a better distribution for other users. Thanks

[Stachelbeck]

Sent you a seperate e-mail with files attached.

Jerome,

The patch we are referring to may not be the correct answer. It might be the results of inappropriate definition in my AADL files.

However here is how it all started.

After down loading, building and installing your change for the assertion failure here is the chain of events.

These changes cleared the assertion failure, Code was generated without error but, during the compiling and linking of the generated code and user code I received undefined references which we have previously discussed before and you had referred me to the producer-consumer example. I extracted the network structure and with a few iterations I was able to get all of my process to compile and link.

The processes m0rt, m1rt and m2rt were designed to pass a simple message to the ewbc and report when it sent its message. The ewbc process was design to receive these messages and report when the message was received.

The first test showed no messages were being sent or received. So I started to debug to determine why I was not getting any messages being sent and received. So starting with main.c begain debugging with po_hi_driver_sockets_init in po_hi_driver_sockets.c by turning on debug in this file.

The debug trace showed ewbc, m0rt, m1rt and m2rt were connecting to IP address 127.0.0.1 and port 0.

I followed backwards in the code to see how the port was being defined. It traced back to the deviceconf.c file which was defined as (per your example) :

#include <drivers/configuration/ip.h

__po_hi_c_ip_conf_t pohidrv_l0 = { "l0", "127.0.0.1", __po_hi_c_ipv4, 4000};
__po_hi_c_ip_conf_t pohidrv_l1 = { "l0", "127.0.0.1", __po_hi_c_ipv4, 4001};
__po_hi_c_ip_conf_t pohidrv_l2 = { "l0", "127.0.0.1", __po_hi_c_ipv4, 4002};
__po_hi_c_ip_conf_t pohidrv_l3 = { "l0", "127.0.0.1", __po_hi_c_ipv4, 4003};

Looking at the ip.h file the above initialization did not fit the structure __po_hi_c_ip_conf_t. So I change the devicesconf.c as follows:

#include <drivers/configuration/ip.h>

__po_hi_c_ip_conf_t pohidrv_l0 = {"lo","127.0.0.1","","","","",__po_hi_c_ipv4, 4000,{0,0,0,0,0}};
__po_hi_c_ip_conf_t pohidrv_l1 = {"lo","127.0.0.1","","","","",__po_hi_c_ipv4, 4001,{0,0,0,0,0}};
__po_hi_c_ip_conf_t pohidrv_l2 = {"lo","127.0.0.1","","","","",__po_hi_c_ipv4, 4002,{0,0,0,0,0}};
__po_hi_c_ip_conf_t pohidrv_l3 = {"lo","127.0.0.1","","","","",__po_hi_c_ipv4, 4003,{0,0,0,0,0}};

Re-ran my test. This time debug showed each process connecting to 127.0.0.1 and ewbc to port 4000, m0rt to port 4001, m1rt to port 4002 and m2rt to port 4003 and claiming connections were made. However, no message were still being sent. Reviewing the debug trace once again I found in each of process one the binding on a receive socket being denied with an error code of 98 (Address already in use). So after examining po_hi_driver_sockets_init again and looking at main.c I saw this method was being called twice I came to the conclusion that the second call was not needed. So I commented this line out in each of the main.c for all of the processes.

__PO_HI_MAIN_TYPE __PO_HI_MAIN_NAME (void)
{

/*!

• \var period

• \brief Variable for task period
  *

• This variable is used to store the valueof the period of a task when we

  • create it. The value put in the variable is set according to AADL model
  • description
    */
    __po_hi_time_t period;

  __po_hi_initialize_early ();
  __po_hi_driver_sockets_init (l0_device_id);

/*!

• Initialize the runtime
  _/
  __po_hi_initialize ();
  /___po_hi_driver_sockets_init (0);*/

/*!

• Store the period time for task bcinit
  */
  __po_hi_seconds (&(period), 0);

Then rebuilt and re-ran my test. This time I was now getting connections and each of the processes were reporting the sending and receiving of messages.

However, reviewing the audit trail for messages being sent and received I notice that number of messages sent would increase at a steady rate but, the received message count would lag then catch up and this action would repeat. Since you are creating the socket interface as TCP protocol this action reminded me of the Nagle’s algorithm and since real-time processes need messages to be sent on time you need to set the socket option TCP_NODELAY to remove this algorithm. So I added the following code in the section of po_hi_driver_sockets_init where you are creating the sockets to send message on.

(Starting about line number 565 in po_hi_driver_sockets.c )

  while (1)
  {
     __po_hi_c_sockets_write_sockets[dev] = socket (AF_INET, SOCK_STREAM, 0);

     if (__po_hi_c_sockets_write_sockets[dev] == -1 )
     {
        __DEBUGMSG ("[DRIVER SOCKETS] Socket for dev %d is not created\n", dev);
        return;
     }

    int NoDelayFlag = 1;
    if(setsockopt(__po_hi_c_sockets_write_sockets[dev],IPPROTO_TCP,TCP_NODELAY,&NoDelayFlag,sizeof(NoDelayFlag))){
        __DEBUGMSG ("[DRIVER SOCKETS] Unable to set TCP_NODELAY for dev %d\n", dev);
    }

     __DEBUGMSG ("[DRIVER SOCKETS] Socket for dev %d created, value=%d\n", dev, __po_hi_c_sockets_write_sockets[dev]);

     hostinfo = NULL;

Also, added at the top of file:

#include <netinet/tcp.h>

Now with this change my send and receive message counts tracked each other.

So, why does ocarina generate the second call to po_hi_driver_sockets_init with a value of 0?

Could it be because I’m missing a AADL definition?

I’ve attached all of my AADL files and source files to this e-mail for your review.

My changes only apply to LINUX and LINUX64 environments which is what I’m currently working with.

Also I attached AADL_Project.aadl since I modified it to include ASN1 in supported languages.

Thanks,

Kenneth Stachelbeck
GTRI

[Stachelbeck]

Sorry. This system must delete certain character marking. the include should be including netinet/tcp.h.

[yoogx]

OK, I nailed down the bug, this is the same artefact than the assertion failure on the location: the process is bound to a virtual processor and not a processor, this bugs a little bit the code generator.

Patch is straightforward, I'll update the generator accordingly.

[yoogx]

Regarding the generation of the second call to po_hi_driver_sockets_init with a value of 0: it is because of the definition of the ethernet bus, you have to remove the following line
Deployment::Transport_API => BSD_Sockets;

This line triggers a circuitry that is looking for the "old" configuration style

I also committed your patch for the TCP_NODELAY option in PolyORB-HI/C.

The latest step would be to introduce code patterns for attaching process and threads to OS cores. I think you have a patch for that, can you provide me some hints to integrate it? Thanks

[Stachelbeck]

I was not aware of the association of “Depoyment::Transport_API => BSD_Sockets;” to the second po_hi_driver_sockets_init since I do not know how ocarina generates code with respect to AADL. All I know is from debugging the second call caused an error which prevented communications between processes and by commenting out the second call to po_hi_driver_sockets_init in each of the generated main.c’s this it removed the error which then allowed for communications. When I get home I’ll remove the “Deployment::Transport_API => BSD_Sockets;” and test to see if I gets me what is needed to run and communicate without modifying the main.c’s. I’ll send you the results of my testing.

[yoogx]

Note that I also uploaded your model to AADLib repository

About the organization of the patterns, there are rooms for improvement and documentation, obviously ;)

[Stachelbeck]

Is it possible for me to make changes to the model I sent you and you just up loaded to the AADL Libraries?

[yoogx]

Sure, you can either send me the new version, or perform a "pull" request on the repository.

[Stachelbeck]

Jerome,

Thanks.

[yoogx]

Actually this has now been supported for a while, closing this ticket.