[OT] Encode woes

[Dirk Koopman]

Dirk Koopman wrote:
> It appears that, with the increasing prevalence of 5.10, the usage of 
> utf8 or not is getting more picky.
> 
> I have a well established, networked, app that has upwards of 250 nodes 
> and about 4000 users at one time (on certain weekends double that) all 
> over the world. These users are running mainly windows based clients 
> (which may include quite a lot of windows telnet). The nominal character 
> set is ascii, as interpreted by the client's host operating system.
> 
> To date, I have managed to avoid the tribulations of Encode and utf8 et 
> al. But I am now get occasional errors, on 5.10 perl, of the ilk:-
> 
>  Wide character in null operation at /spider/perl/DXDupe.pm line 47.
>  at /spider/perl/DXDupe.pm line 47
>     DXDupe::find('X14163|UA0KEF|RZ6HV|�������  �������') called at 
> /spider/perl/Spot.pm line 420
> 
> And also something similar on print or syswrite.
> 
> Studying the data, what I am receiving is a mixture of utf8 and 
> iso-8859-*, the reason for this being that older perls happily take what 
> they are given and just pass it along. Some clients are emitting utf8 
> and other iso-8859 and yet others (running Win95/8) some kind of 
> codepage. In addition, there are older, usually windows based, packages 
> acting as nodes, together with yet more clients that are also adding 
> data to this network in who knows what character set.
> 
> Up until recently, this has not been a problem because the important 
> stuff is in 7 bit ascii and the remarks section (the usual source of 
> problems), if it is unreadable, doesn't matter 'cos you can't translate 
> it anyway.
> 
> Now, is there a reasonably reliable way of determining what we have, on 
> a string by string basis, to at least tell whether we are dealing with 
> utf8 or iso-8859 (not caring which variant) so that I can drive Encode 
> appropriately to avoid crashes of the above type.  Or how do I 
> completely switch off utf8 encoding/decoding - everywhere - in an 80,000 
> line perl app.
> 

As no-one seems interested in this, or may be no-one else has had these 
problems themselves, can anyone suggest a better mailing list to poll?

Dirk