error/exception aggregation during a "big" dbic load

Bob MacCallum uncoolbob at
Tue May 8 23:30:53 BST 2012

On Tue, May 8, 2012 at 11:13 PM, Yan Fitterer <yan at> wrote:

> One obvious approach would be to validate the data in a separate step.
Yes, this had occurred to me, and there is even a (Java) validator tool for
the input files we are using which might work out for us (I haven't looked
at it in detail).  But it seems like writing a validator and a loader would
duplicate a lot of effort and be difficult to maintain, but maybe I'm

Thanks for your reply - obvious isn't always obvious to people like me
asking these questions.

> Sent from my iPhone
> On 8 May 2012, at 22:03, Bob MacCallum <uncoolbob at> wrote:
> > Hello - I would greatly appreciate any suggestions for how to keep track
> of
> > errors while loading a big chunk of data into a dbic schema (in a
> > txn_do(...), from a file).
> >
> > The problem is that I don't want to throw an exception as soon as I
> > encounter the first error (for example a user record missing its required
> > email field) - I would rather create some placeholder data and carry on
> > loading stuff (for example loading objects that belong to this user) so
> > that I can dump a list of the errors right at the end before rolling back
> > the transaction.  The reason I want this is because the input file is
> > provided by a third party and it would reduce the number of emails back
> and
> > forth if we can send them as many corrections as possible at one time.
> >
> > Being a bioinformaticist and not a pro coder, I didn't know what to
> google
> > for (I tried what I could) so apologies if the answer's out there on
> > I can't be the only person who's ever wanted to do this.
> >
> > thanks,
> > Bob.

More information about the mailing list