error/exception aggregation during a "big" dbic load

[Bob MacCallum]

On Tue, May 8, 2012 at 11:13 PM, Yan Fitterer <yan at fitterer.org> wrote:

> One obvious approach would be to validate the data in a separate step.
>
>
Yes, this had occurred to me, and there is even a (Java) validator tool for
the input files we are using which might work out for us (I haven't looked
at it in detail).  But it seems like writing a validator and a loader would
duplicate a lot of effort and be difficult to maintain, but maybe I'm
mistaken.

Thanks for your reply - obvious isn't always obvious to people like me
asking these questions.


> Sent from my iPhone
>
> On 8 May 2012, at 22:03, Bob MacCallum <uncoolbob at gmail.com> wrote:
>
> > Hello - I would greatly appreciate any suggestions for how to keep track
> of
> > errors while loading a big chunk of data into a dbic schema (in a
> > txn_do(...), from a file).
> >
> > The problem is that I don't want to throw an exception as soon as I
> > encounter the first error (for example a user record missing its required
> > email field) - I would rather create some placeholder data and carry on
> > loading stuff (for example loading objects that belong to this user) so
> > that I can dump a list of the errors right at the end before rolling back
> > the transaction.  The reason I want this is because the input file is
> > provided by a third party and it would reduce the number of emails back
> and
> > forth if we can send them as many corrections as possible at one time.
> >
> > Being a bioinformaticist and not a pro coder, I didn't know what to
> google
> > for (I tried what I could) so apologies if the answer's out there on
> CPAN.
> > I can't be the only person who's ever wanted to do this.
> >
> > thanks,
> > Bob.
>