shsim.h is the header file for shared memory, and dsim.h is the header file for distributed memory.

#ifdef DSIM_SHMEM
#include "../../src/shsim.h"
#else
#include "../../src/dsim.h"
#endif

These are several variable used by the PHOLD simulation. X is the number of LPs per row, Y is the number of LPs per column, and N is the number of events initially assigned to each LP.

unsigned int X,Y,N;
simtime_t Delay;

Any LP must be derived from the base class tw_lp

class PHold : public tw_lp
{
 public:

This statement defines an event type. The first parameter, unsigned int specifies the data type of the positive event. The second parameter, unsigned long long specifies the data type of the anti-event. The third parameter is the type id of the event type. If an LP has multiple event types, their type ids should be different in order to distinguish them.

    typedef tw_event_t<unsigned int,unsigned long long,0> event_t;

Start() is called when the simulation is starting. Stop() is called when the simulation is stopped.

    void Start();
    void Stop();
 protected:

Process() is called to process an event, and Undo() is called to undo an event. Another method that does not appear in this example is Commit(), which is called when the event is being fossil-collected.

    void Process(tw_event*);
    void Undo(tw_event*);
};

In the Start() function, we create N initial events.

void PHold::Start()
{
    unsigned int i;
    for(i=0;i<N;i++)
    {

This function schedules an event. The type of the event being scheduled is event_t. The first argument is the id of the LP that the event will be sent to; here the LP schedules the event for itself, so GetID() is used to obtain its own LP id. The second argument is the positive data of the event; we still put GetID() here because we save the LP id to the event and later when we receive the event we can compare to see the validity of the event. The third argument is the absolute timestamp of the event (notice that the time is not relative to the current simulated time. Each LP has its own random number generator, and Random() is a uniform random number function.

	Schedule<event_t>(GetID(),GetID(),Random(Delay));
    }
}

We do nothing in the Stop() function.

void PHold::Stop()
{
}

In this function we perform the forward computation. Notice that there only one event type in this example, so whenever there is an event we are sure it must belong to the type event_t. In case of multiple event types within one LP, e->type_id (which is the third parameter used to define an event type) returns the type id of the event and it can help you determine the type of the arriving event.

void PHold::Process(tw_event* e)
{

e is of type tw_event, so we need to convert it to event_t.

    event_t* event = static_cast<event_t*>(e);

The positive data of the event contains the id of the LP that should receive the event, so here we check if it is equal to the id returned by GetID().

    assert(event->pe==GetID());

Now we store the seed of the random number generator in this LP, returned by GetSeed(), to the anti-data of the event. The purpose is to make sure we are able the restore the random number generator when the event needs to be rolled back.

    event->ae=GetSeed();

    int i,n;

You can skip this part. What it says is, when RND_EVENTS is defined, n (the number of events scheduled) will be either 0, 1, or 2, so the number of events may be up and down, while the average remains the same. If RND_EVENT is not defined, n is always 1.

#ifdef RND_EVENTS
    n=0;
    if(Random()>0.5)n++;
    if(Random()>0.5)n++;
#else
    n=1;
#endif
    for(i=0;i<n;i++)
    {

We need to find the destination LP for the event to be scheduled. With equal probabilities, this LP could be one of four neighboring LPs.

	int id=GetID();
	double r=Random();
	if(r<0.25)
	    id--;
	else if(r<0.5)
	    id-=Y;
	else if(r<0.75)
	    id+=Y;
	else
	    id++;
	if(id<0)id+=X*Y;
	if(id>=(int)(X*Y))id-=X*Y;

Now we schedule the event. Remember the timestamp of the event is in absolute simulated time, so we call SimTime() to obtain the current simulated time, and then add a random time returned by Exponential().

	Schedule<event_t>(id,id,SimTime()+Exponential(Delay));
    }
}

In this function we perform the reverse computation.

void PHold::Undo(tw_event* e)
{
    event_t* event= static_cast<event_t*>(e);

Here we restore the seed of the random number generator in this LP.

    SetSeed(event->ae);

Here we restore the positive data of the event. Why? because, to save memory, an event may use the same memory block to store both positive data and anti-data (determined by a macro DSIM_EVENT_UNION).

    event->pe=GetID();
}

This is the array of LPs.

PHold* phold_lps;

lp_start is the id of the first LP in the processor, and lp_total is the total number of LP in this processor.

unsigned int lp_start;
unsigned int lp_total;

This function is required. It returns the id of the processor that the LP with the specified id resides.

inline unsigned int id_to_proc(unsigned int id)
{

The mappings from LPs to processors are different in two modes. tw_procs is the total number of processors.

#ifdef DSIM_SHMEM
  return id*tw_procs/lp_total;
#else
    return id/lp_total;
#endif
}

Another required function. It returns the address of the LP with the specified id.

inline tw_lp* id_to_lp ( unsigned int id)
{
    assert(id>=lp_start && id<lp_start+lp_total);
    return &phold_lps[id-lp_start];
}

Now we need to provide a tw_main function.

int tw_main(int argc,char** argv)
{

Initiate a simulation engine first. StopTime is the simulated time at which the simulation stops. EventBatch is the number of events processed before the simulation engine looks at its input event queue (contains events sent by other processors). GVTInterval is the time elapses before reporting to the GVT master.

    tw_simeng se;
    se.StopTime=500;
    se.EventBatch=16;
    se.GVTInterval=0.1;

Set the parameters with default values

    X=8192;
    Y=16;
    N=16;
    Delay=10;

Reset parameters if they are input from the command line

    if(argc>1) X=atoi(argv[1]);
    if(argc>2) se.StopTime=atoi(argv[2]);
    if(argc>3) se.GVTInterval=atof(argv[3]);
    if(argc>4) se.EventBatch=atoi(argv[4]);

Set lp_total and lp_start.

#ifdef DSIM_SHMEM
    lp_total=X*Y;
    lp_start=0;
#else
    lp_total=X*Y/tw_procs;
    lp_start=tw_rank*X*Y/tw_procs;
#endif

    if((X*Y%tw_procs)!=0) return 1;

Instantiate the LPs.

    phold_lps=new PHold[lp_total];
    for(unsigned int i=0;i<lp_total;i++)
    {

The Init() function must be called for every LP. This function takes as arguments the address of the simulation engine, the seed, and the LP id. The seed for all LPs is the same here, for each LP will adjust the seed value by adding its id to it.

	phold_lps[i].Init(&se,1,lp_start+i);
    }

Now we can run the simulation

    se.Run();

Don't forget to release the memory before the program exits.

    delete [] phold_lps;

    return 0;
}