Adapt dictionary compilation to new default encoding

The problem is that the change in default encoding to utf8 in 17.0, in
commit 00e42967, changes the behaviour of erl_parse:abstract/1, which is
used by the dictionary compiler to turn terms into abstract code. In
particular, it transforms the orddict representation of a parsed
dictionary to contruct the return value of a dictionary module's dict/0
function. This orddict contains various lists, one of which is a list of
tuples of the form

  {Name, Code, [VendorId], [Avp]}

where Name is an ASCII string and VendorId is a non-negative integer.

Using erl_parse:abstract/2 instead allows a string encoding to be
specified but regardless of what encoding is used, the result of
transforming our tuple might not be what we really want, which is for
Name to always be represented as a string form and [VendorId] to always
be represented as a cons form: the [VendorId] will always end up as a
string form if the integers are small enough. The only way around this
is to transform the tuple bit by bit, but modifying the code to do this
is quite a lot of work, for not much gain: it would be nice to produce
more readable output but nothing stops working without it.

This commit restores the pre-17.0 conversion by explicilty specifying
latin1 as the string encoding to erl_parse:abstract/2. The utf8 encoding
broke the compilation of some dictionares since unicode strings aren't
expected when writing the generated code to file.

Note that the latin1 encoding does reasonably well in practice, although
it mangles the Ericsson Vendor Id list [193] into a "LATIN CAPITAL
LETTER A WITH ACUTE". The utf8 encoding does worse, mangling the 3GPP
Vendor Id 10415 into "DESERET CAPITAL LETTER CHEE". An ascii encoding
would do better than latin1 but doesn't yet exist. (Encoding isn't
really what the option is. It's a string predicate: if the predicate is
true then represent as a string form, otherwise a cons form.)

2 files changed, 12 insertions, 2 deletions

diff --git a/lib/diameter/src/compiler/diameter_forms.hrl b/lib/diameter/src/compiler/diameter_forms.hrl
index 9b14c1715a..da680666f0 100644
--- a/lib/diameter/src/compiler/diameter_forms.hrl
+++ b/lib/diameter/src/compiler/diameter_forms.hrl
@@ -1,7 +1,7 @@
 %%
 %% %CopyrightBegin%
 %%
-%% Copyright Ericsson AB 2010-2013. All Rights Reserved.
+%% Copyright Ericsson AB 2010-2014. All Rights Reserved.
 %%
 %% The contents of this file are subject to the Erlang Public License,
 %% Version 1.1, (the "License"); you may not use this file except in
@@ -57,4 +57,4 @@
 -define(FIELDS(Fs),   [{?record_field, ?ATOM(F), V} || {F,V} <- Fs]).
 
 %% Literal term.
--define(TERM(T),      erl_parse:abstract(T, ?LINE)).
+-define(TERM(T), erl_parse:abstract(T, [{line, ?LINE}, {encoding, latin1}])).
diff --git a/lib/diameter/test/diameter_compiler_SUITE.erl b/lib/diameter/test/diameter_compiler_SUITE.erl
index df4dde6240..08ffe5981d 100644
--- a/lib/diameter/test/diameter_compiler_SUITE.erl
+++ b/lib/diameter/test/diameter_compiler_SUITE.erl
@@ -120,6 +120,16 @@
          {avp_has_duplicate_flag,
           " -",
           " MM"},
+         {ok,
+          "@vendor 0",
+          "@vendor 10415"},
+         {ok,
+          [{"@vendor 0", "@vendor 10415"},
+           {"Proxy-Info .*M$", "&V"},
+           {"Proxy-Info ::= [^>]*", "& 10415 "}]},
+         {grouped_vendor_id_without_flag,
+          [{"@vendor 0", "@vendor 10415"},
+           {"Proxy-Info ::= [^>]*", "& 10415 "}]},
          {avp_has_vendor_id,
           "@avp_types",
           "@avp_vendor_id 667 Class\n&"},