Best practice for variables shared between processes?

[Peter Flanigan]

When implementing the producer/consumer pattern within a file system I 
use these guidelines.

Pick a directory writeable by the producer, readable by the consumer 
(this method only uses one directory to avoid file system boundary 
issues). Have the producer write a file with a name that matches this 
template:

	B_<group_prefix><sequence_number>.<suffix>

Call it the "before" image. When the write is complete, rename the file 
removing the B_ prefix. The consumer polls for files matching the pattern:

	<group_prefix>[0-9]+\.<suffix>

The group prefix allows for multiple streams of files to coexist in the 
same directory. Useful if the producer is on a remote host since only 
one identity and writeable directory has to set up to enable multiple 
streams.

If the sequence number is chosen to be fixed length and ascending then 
it defines the processing order. After the consumer has processed the 
file have it rename the file prefixing it with A_, the "after" image.

Select the suffix to be association table friendly for clicking on the 
files in a GUI environment. The use of prefixes rather than appending a 
".tmp" suffix for the before image leaves all three file states with the 
correct suffix for the association table.

This approach has a number of useful properties.

The state is given by; ls -1. It's output groups the before, during and 
after images. Within each group file are listed in processing order.

Reprocessing one or more files is achieved by renaming them, striping 
the A_ prefix (much easier than "touching" a list of files with pauses 
which is what happens if you use mtime).

Before image files that are "old" should be investigated and deleted 
since their continued existence indicates an error. After image files 
that are "old" can archived/deleted. Files without prefixes that become 
"old" indicate that the consumer has stopped processing.

Works using FTP, ssh/scp, local processes or anything that has a 
functional rename command and is dependant only on the atomicity of that 
command.

-- 

Regards